From 2e79ccfeb5130dfaad63627754b12527b92f625d Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Wed, 14 Oct 2026 19:04:20 -0700 Subject: [PATCH 1/5] docs(coding-agents): Pi Coding Agent SDK tracing (TH-8264) Page and runnable example for @traceai/pi-coding-agent: wrap createAgentSession (or instrumentSession) and every session.prompt() becomes one trace (agent run > turns > model calls and tool calls), grouped by the Pi session id. Content is off by default (FI_HIDE_INPUTS/FI_HIDE_OUTPUTS=false opt in). States the tested version (@earendil-works/pi-coding-agent 1.2.0), the npm scope move, what is not available, and that this is not the Pi app or Oh My Pi. Co-Authored-By: Claude Opus 5.5 --- examples/coding-agents/pi-sdk/README.md | 17 ++ .../coding-agents/pi-sdk/models.example.json | 10 ++ examples/coding-agents/pi-sdk/package.json | 12 ++ examples/coding-agents/pi-sdk/run_e2e.sh | 38 +++++ .../coding-agents/pi-sdk/trace_session.mjs | 44 +++++ src/lib/navigation.ts | 1 + .../integrations/coding-agents/pi-sdk.mdx | 156 ++++++++++++++++++ 7 files changed, 278 insertions(+) create mode 100644 examples/coding-agents/pi-sdk/README.md create mode 100644 examples/coding-agents/pi-sdk/models.example.json create mode 100644 examples/coding-agents/pi-sdk/package.json create mode 100755 examples/coding-agents/pi-sdk/run_e2e.sh create mode 100644 examples/coding-agents/pi-sdk/trace_session.mjs create mode 100644 src/pages/docs/integrations/coding-agents/pi-sdk.mdx diff --git a/examples/coding-agents/pi-sdk/README.md b/examples/coding-agents/pi-sdk/README.md new file mode 100644 index 000000000..dd19554a5 --- /dev/null +++ b/examples/coding-agents/pi-sdk/README.md @@ -0,0 +1,17 @@ +# Pi Coding Agent SDK tracing + +Runnable example for [Pi Coding Agent SDK tracing](https://docs.futureagi.com/docs/integrations/coding-agents/pi-sdk): one `createAgentSession()` session, traced by [`@traceai/pi-coding-agent`](https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent) and exported to Future AGI. + +| File | What it is | +| --- | --- | +| `trace_session.mjs` | Creates a traced Pi session, runs one prompt, flushes spans | +| `models.example.json` | Pi `models.json` for an OpenAI-compatible endpoint | +| `run_e2e.sh` | Isolated end-to-end run with a throwaway project and a privacy marker | +| `package.json` | Pinned to the tested Pi SDK (1.2.0) | + +```bash +npm install +FI_API_KEY=... FI_SECRET_KEY=... FI_PROJECT_NAME=pi-sdk node trace_session.mjs "List the files here" +``` + +Content (prompts, tool arguments, tool results, answers) is off by default. Set `FI_HIDE_INPUTS=false` and/or `FI_HIDE_OUTPUTS=false` to include it. diff --git a/examples/coding-agents/pi-sdk/models.example.json b/examples/coding-agents/pi-sdk/models.example.json new file mode 100644 index 000000000..c85c55a71 --- /dev/null +++ b/examples/coding-agents/pi-sdk/models.example.json @@ -0,0 +1,10 @@ +{ + "providers": { + "my-gateway": { + "baseUrl": "http://127.0.0.1:8317/v1", + "api": "openai-completions", + "apiKey": "$MY_GATEWAY_API_KEY", + "models": [{ "id": "gpt-6-luna" }] + } + } +} diff --git a/examples/coding-agents/pi-sdk/package.json b/examples/coding-agents/pi-sdk/package.json new file mode 100644 index 000000000..cf11855ed --- /dev/null +++ b/examples/coding-agents/pi-sdk/package.json @@ -0,0 +1,12 @@ +{ + "name": "futureagi-pi-sdk-tracing-example", + "private": true, + "type": "module", + "description": "Trace a Pi Coding Agent SDK session in Future AGI with @traceai/pi-coding-agent", + "engines": { "node": ">=22.19.0" }, + "dependencies": { + "@earendil-works/pi-coding-agent": "1.2.0", + "@traceai/fi-core": "^1.0.0", + "@traceai/pi-coding-agent": "^0.1.0" + } +} diff --git a/examples/coding-agents/pi-sdk/run_e2e.sh b/examples/coding-agents/pi-sdk/run_e2e.sh new file mode 100755 index 000000000..71e418e21 --- /dev/null +++ b/examples/coding-agents/pi-sdk/run_e2e.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# End-to-end check: one real Pi SDK session, traced into a Future AGI collector. +# +# FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=http://localhost:14338 \ +# PI_PROVIDER=my-gateway PI_MODEL=gpt-6-luna MY_GATEWAY_API_KEY=... \ +# ./run_e2e.sh [workdir] +# +# Creates an isolated Pi agent dir (models.json from $PI_MODELS_JSON, default +# models.example.json) and a throwaway project with a README.md, installs the +# example, and runs one prompt that uses the ls and read tools. The prompt +# carries a unique marker: with the default settings the marker must NOT reach +# Future AGI; rerun with FI_HIDE_INPUTS=false to see it on the run span. +# TRACEAI_PI_PACKAGE may point at a local tarball (absolute path) of +# @traceai/pi-coding-agent, for example one built with `pnpm pack`. +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +WORK="${1:-$(mktemp -d)}" +: "${FI_API_KEY:?set FI_API_KEY}" "${FI_SECRET_KEY:?set FI_SECRET_KEY}" +: "${PI_PROVIDER:?set PI_PROVIDER}" "${PI_MODEL:?set PI_MODEL}" +export FI_PROJECT_NAME="${FI_PROJECT_NAME:-pi-sdk-e2e}" + +mkdir -p "$WORK/app" "$WORK/agent" "$WORK/project" +cp "$HERE/package.json" "$HERE/trace_session.mjs" "$WORK/app/" +cp "${PI_MODELS_JSON:-$HERE/models.example.json}" "$WORK/agent/models.json" +printf '# Throwaway project\n\nCreated by run_e2e.sh.\n' > "$WORK/project/README.md" + +cd "$WORK/app" +if [ -n "${TRACEAI_PI_PACKAGE:-}" ]; then + npm pkg set "dependencies.@traceai/pi-coding-agent=file:$TRACEAI_PI_PACKAGE" +fi +npm install --ignore-scripts --silent + +MARKER="PI-SDK-MARKER-$(date +%s)-$$" +PI_CODING_AGENT_DIR="$WORK/agent" PI_CWD="$WORK/project" \ + node trace_session.mjs "List the files in this folder, then read README.md and give me its title. $MARKER" +echo "marker=$MARKER" +echo "project=$FI_PROJECT_NAME" diff --git a/examples/coding-agents/pi-sdk/trace_session.mjs b/examples/coding-agents/pi-sdk/trace_session.mjs new file mode 100644 index 000000000..c91b90b28 --- /dev/null +++ b/examples/coding-agents/pi-sdk/trace_session.mjs @@ -0,0 +1,44 @@ +// Run one Pi Coding Agent SDK session and trace it in Future AGI. +// +// node trace_session.mjs "" +// +// Environment: +// FI_API_KEY, FI_SECRET_KEY Future AGI keys (sent as X-Api-Key / X-Secret-Key) +// FI_PROJECT_NAME Future AGI project (default: pi-sdk) +// FI_BASE_URL collector base URL (default: https://api.futureagi.com) +// FI_HIDE_INPUTS=false opt in to prompts and tool arguments +// FI_HIDE_OUTPUTS=false opt in to answers and tool results +// PI_CODING_AGENT_DIR Pi agent dir with models.json / auth.json (default: ~/.pi/agent) +// PI_PROVIDER, PI_MODEL model to use (default: Pi's configured default) +// PI_CWD project directory the agent works in (default: cwd) +import { ProjectType, register } from "@traceai/fi-core"; +import { PiCodingAgentInstrumentation } from "@traceai/pi-coding-agent"; +import { createAgentSession, ModelRuntime, SessionManager } from "@earendil-works/pi-coding-agent"; + +const prompt = process.argv[2] ?? "List the files in this folder and read README.md."; +const cwd = process.env.PI_CWD ?? process.cwd(); + +const tracerProvider = register({ + projectType: ProjectType.OBSERVE, + projectName: process.env.FI_PROJECT_NAME ?? "pi-sdk", +}); +const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider }); +const createTracedSession = instrumentation.wrapCreateAgentSession(createAgentSession); + +const options = { cwd, sessionManager: SessionManager.inMemory(cwd), tools: ["read", "ls", "bash"] }; +if (process.env.PI_PROVIDER && process.env.PI_MODEL) { + const modelRuntime = await ModelRuntime.create(); + const model = modelRuntime.getModel(process.env.PI_PROVIDER, process.env.PI_MODEL); + if (!model) throw new Error(`Pi does not know ${process.env.PI_PROVIDER}/${process.env.PI_MODEL}; add it to models.json`); + Object.assign(options, { model, modelRuntime }); +} + +const { session } = await createTracedSession(options); +try { + await session.prompt(prompt); + console.log(session.getLastAssistantText()); + console.error(`pi session id: ${session.sessionId}`); +} finally { + session.dispose(); + await tracerProvider.shutdown(); +} diff --git a/src/lib/navigation.ts b/src/lib/navigation.ts index 9ebda640c..942ff48e4 100644 --- a/src/lib/navigation.ts +++ b/src/lib/navigation.ts @@ -980,6 +980,7 @@ export const tabNavigation: NavTab[] = [ // coding-agents:opencode // coding-agents:pi // coding-agents:pi-sdk + { title: 'Pi SDK', href: '/docs/integrations/coding-agents/pi-sdk' }, ] }, { diff --git a/src/pages/docs/integrations/coding-agents/pi-sdk.mdx b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx new file mode 100644 index 000000000..6af37f783 --- /dev/null +++ b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx @@ -0,0 +1,156 @@ +--- +title: "Pi Coding Agent SDK tracing with Future AGI" +description: "Trace Pi Coding Agent SDK sessions (createAgentSession) in Future AGI with @traceai/pi-coding-agent: one trace per agent run, with turns, model calls, tokens, tool calls, and errors. Prompt and code content stay off by default." +--- + +## About + +The [Pi Coding Agent SDK](https://pi.dev/docs/latest/sdk) embeds the Pi coding agent in a Node.js process: `createAgentSession()` returns an `AgentSession` that you prompt from your own code, and `session.subscribe()` streams its run, turn, message, and tool events. The `@traceai/pi-coding-agent` package subscribes to those events and exports them to Future AGI, so every `session.prompt()` shows up as one trace: + +``` +invoke_agent pi one agent run (session.prompt), session.id = Pi session id +├── pi.turn one model response and the tools it called +│ ├── chat gpt-6-luna model call: model, provider, tokens, cost, finish reason +│ └── execute_tool ls tool call: name, call id, duration, error flag +├── pi.turn +│ ├── chat gpt-6-luna +│ └── execute_tool read +└── pi.turn + └── chat gpt-6-luna final answer +``` + +This is not the same as: + +- **Pi, the command-line app** (`pi`). Interactive Pi sessions are traced with a Pi extension, not with this package. See the Pi page in this section. +- **Oh My Pi** (`omp`, the omp.sh fork). It has its own hook API and its own page. +- **Model-provider tracing.** `@traceai/openai` or `@traceai/anthropic` trace raw provider calls. Pi makes its model calls itself; running a provider instrumentor in the same process adds duplicate model spans. + +| | | +| --- | --- | +| Tested with | `@earendil-works/pi-coding-agent` 1.2.0, Node 26, `@traceai/pi-coding-agent` 0.1.0 | +| Pi docs used | [SDK](https://pi.dev/docs/latest/sdk), [Models](https://pi.dev/docs/latest/models), `AgentSessionEvent` in the 1.2.0 type declarations, read 2026-10-14 | +| Requires | Node.js 22.19 or newer (Pi's own requirement) | +| Source | [`@traceai/pi-coding-agent`](https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent), [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk) | + + +Pi moved from the `@mariozechner/*` npm scope to `@earendil-works/*` at version 0.74.0. Install `@earendil-works/pi-coding-agent`. The tracing package reads the session object by shape and does not depend on either scope. + + +## 1. Install + +```bash +npm install @traceai/pi-coding-agent @traceai/fi-core @earendil-works/pi-coding-agent +``` + +## 2. Add your Future AGI keys + +```bash .env +FI_API_KEY=your-futureagi-api-key +FI_SECRET_KEY=your-futureagi-secret-key +FI_PROJECT_NAME=my-pi-agent +``` + +`@traceai/fi-core` reads these and sends the keys as the `X-Api-Key` / `X-Secret-Key` headers. They are never written onto spans. For a self-hosted Future AGI, also set `FI_BASE_URL` to your collector, for example `http://localhost:14338`. + +## 3. Trace your sessions + +The Pi SDK is ESM-only, so its exports cannot be patched in place. Wrap `createAgentSession` once and create sessions through the wrapper: + +```typescript +import { register, ProjectType } from "@traceai/fi-core"; +import { createAgentSession, SessionManager } from "@earendil-works/pi-coding-agent"; +import { PiCodingAgentInstrumentation } from "@traceai/pi-coding-agent"; + +const tracerProvider = register({ projectType: ProjectType.OBSERVE }); +const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider }); +const createTracedSession = instrumentation.wrapCreateAgentSession(createAgentSession); + +const { session } = await createTracedSession({ sessionManager: SessionManager.inMemory() }); +try { + await session.prompt("List the files in this folder and read README.md"); +} finally { + session.dispose(); // ends any span still open + await tracerProvider.shutdown(); // flush before the process exits +} +``` + +If something else creates the session (for example an `AgentSessionRuntime`), trace it directly before you prompt it: + +```typescript +const stop = instrumentation.instrumentSession(session); // stop() ends tracing for that session +``` + +An `AgentSessionRuntime` creates a new `AgentSession` on new, resume, fork, and import. Call `instrumentSession(runtime.session)` again after each replacement, as Pi's docs require for any subscription. + +The model and credentials come from Pi as usual: `~/.pi/agent/models.json` and `auth.json`, or `PI_CODING_AGENT_DIR`. The tracing does not change which endpoint Pi calls. + +## 4. Run it + +A runnable copy is in [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk). `run_e2e.sh` creates an isolated Pi agent directory and a throwaway project, then runs one prompt that uses the `ls` and `read` tools: + +```bash +FI_API_KEY=... FI_SECRET_KEY=... FI_PROJECT_NAME=pi-sdk-e2e \ +PI_PROVIDER=my-gateway PI_MODEL=gpt-6-luna MY_GATEWAY_API_KEY=... \ +examples/coding-agents/pi-sdk/run_e2e.sh +``` + +Open the project in **Observe**. Each run is one trace; all runs of a session share the same `session.id` and group under **Sessions**. + +## What is captured + +| Pi SDK event | Future AGI span | Fields | +| --- | --- | --- | +| `agent_start` .. `agent_settled` | **invoke_agent pi** (`AGENT`) | `session.id`, `gen_ai.conversation.id`, `pi.session.name`; run totals (`pi.agent.attempts`, turn, model-call and tool-call counts, `pi.agent.usage.*`, `pi.agent.cost.total`); retry and compaction events | +| `turn_start` .. `turn_end` | **pi.turn** (`CHAIN`) | turn index, stop reason, tool-result count | +| assistant `message_start` .. `message_end` | **chat <model>** (`LLM`) | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.provider.name`, `pi.api`, `gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens` / `cache_read_tokens` / `cache_write_tokens`, `gen_ai.cost.total` when Pi priced the call, finish reason, names of the tools it called. Start and end come from the message's own request timestamp and duration | +| `tool_execution_start` .. `tool_execution_end` | **execute_tool <tool>** (`TOOL`) | `gen_ai.tool.name`, `gen_ai.tool.call.id`, duration, `pi.tool.is_error`. A tool called by another tool (`parentToolCallId`) nests under the calling tool | + +Input tokens follow the OpenTelemetry convention and include cached tokens. Pi's own `usage.input` excludes cache reads and writes, so it is kept separately as `pi.usage.uncached_input_tokens`. Run totals use `pi.agent.*` keys, not `gen_ai.usage.*`, so a trace's tokens are not counted twice. + +Errors: + +- A model call that ends with `stopReason` `error` or `aborted` is `ERROR`, with the provider's error message as the status. Its turn is `ERROR` too. +- A run that ends aborted, or whose last model call failed, is `ERROR`. When Pi retries a failed call and the retry succeeds, the run is `OK` and the failed call keeps its own `ERROR` span. +- A tool result with `isError` is `ERROR`. +- Disposing the session mid-run ends the open spans with `pi.incomplete=true`. + +## What is not available + +- **The model request payload.** Session events do not include the transcript Pi sends on each call, so model spans carry no input messages. The user prompt is on the run span when you opt in to content. +- **Streaming detail.** No time to first token and no per-delta events; `message_update` deltas are not traced. +- **Cost for custom models.** Models you add in `models.json` without prices report a cost of 0, and no cost attribute is written. +- **The system prompt.** Never exported, even with content on. +- **Extension-only events.** `before_provider_request`, `before_provider_headers`, and other extension hooks are not part of `session.subscribe()` and are not used. + +## Privacy + +By default only structure and metadata leave your process: session id, model, provider, token counts, finish reasons, tool names and call ids, durations, and statuses. + +- **Not exported by default:** the user prompt, tool arguments, tool results (file contents, command output), assistant text, tool-call arguments, and tool error text. +- **Never exported:** the system prompt and the Pi working directory. + +To include content, opt in with either the instrumentation options or environment variables: + +```typescript +new PiCodingAgentInstrumentation({ tracerProvider, traceConfig: { hideInputs: false, hideOutputs: false } }); +``` + +```bash +FI_HIDE_INPUTS=false # prompt (run span input), tool arguments +FI_HIDE_OUTPUTS=false # answers (model and run span output), tool results, tool error text +``` + +Only the exact value `false` turns capture on; any other value keeps content hidden. An explicit `traceConfig` value wins over the environment. Each value is cut at 16,384 characters (`maxContentLength`). + +## Troubleshooting + +**No trace appears.** +1. `register()` needs a project name (`FI_PROJECT_NAME` or `projectName`) and both keys. +2. A short script must `await tracerProvider.shutdown()` before it exits, or the last spans are lost. +3. The session must be traced before `prompt()`: create it through the wrapper, or call `instrumentSession()` first. + +**Traces stop after `/new`, `/resume`, or a fork.** An `AgentSessionRuntime` replaced the session object. Call `instrumentSession(runtime.session)` again. + +**The run span is not under my request span.** The run is parented to the active span only when your app registered an OpenTelemetry context manager (for example `AsyncLocalStorageContextManager`); otherwise each run starts its own trace. + +**Every model call appears twice.** A provider instrumentor (`@traceai/openai`, `@traceai/anthropic`) is also tracing Pi's HTTP calls in the same process. Remove it, or accept both. From b3bc3b00d6342e21b862a614c0b07468151a0552 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Thu, 15 Oct 2026 08:20:20 -0700 Subject: [PATCH 2/5] docs(coding-agents): Pi SDK review fixes: versions, double traces, error privacy (TH-8264) - Pi versions row: runs end at agent_settled from 0.80.5; 0.75.4-0.80.3 end at agent_end without willRetry; older Pi splits retries. - Note + troubleshooting for double traces when the Pi extension is also installed and the app calls bindExtensions(), with a DefaultResourceLoader extensionsOverride that leaves it out. - Install-from-source steps while @traceai/pi-coding-agent is unreleased (traceAI PR #252). - Privacy: provider/tool/compaction error text, session name and compaction summary are opt-in; working-directory wording; env values are case-insensitive and trimmed. - Compaction span, finish_reasons mapping, pi.agent.attempts/retry_count. - FI_BASE_URL example uses the self-hosted default 4318; .env loading note; register() throws without a project name. - Example: placeholder gateway/model, read+ls tools only, run_e2e.sh accepts relative paths and exits non-zero on a failed run. Co-Authored-By: Claude Opus 5.5 --- examples/coding-agents/pi-sdk/README.md | 16 +++- .../coding-agents/pi-sdk/models.example.json | 4 +- examples/coding-agents/pi-sdk/run_e2e.sh | 35 ++++--- .../coding-agents/pi-sdk/trace_session.mjs | 13 ++- .../integrations/coding-agents/pi-sdk.mdx | 96 ++++++++++++++----- 5 files changed, 120 insertions(+), 44 deletions(-) diff --git a/examples/coding-agents/pi-sdk/README.md b/examples/coding-agents/pi-sdk/README.md index dd19554a5..35e9c262d 100644 --- a/examples/coding-agents/pi-sdk/README.md +++ b/examples/coding-agents/pi-sdk/README.md @@ -5,13 +5,25 @@ Runnable example for [Pi Coding Agent SDK tracing](https://docs.futureagi.com/do | File | What it is | | --- | --- | | `trace_session.mjs` | Creates a traced Pi session, runs one prompt, flushes spans | -| `models.example.json` | Pi `models.json` for an OpenAI-compatible endpoint | +| `models.example.json` | Pi `models.json` for an OpenAI-compatible endpoint, with a placeholder gateway URL and model id | | `run_e2e.sh` | Isolated end-to-end run with a throwaway project and a privacy marker | | `package.json` | Pinned to the tested Pi SDK (1.2.0) | +`@traceai/pi-coding-agent` ships with [traceAI PR #252](https://github.com/future-agi/traceAI/pull/252). Until it is on npm, build its tarball as the [docs page](https://docs.futureagi.com/docs/integrations/coding-agents/pi-sdk) shows (`pnpm pack` in a traceAI checkout of that PR) and pass it to `run_e2e.sh` as `TRACEAI_PI_PACKAGE=/abs/path/traceai-pi-coding-agent-0.1.0.tgz`, or `npm install` it here before the other dependencies. + +Edit `models.example.json` (gateway `baseUrl` and model `id`) or point `PI_MODELS_JSON` at your own Pi `models.json`, then: + +```bash +FI_API_KEY=... FI_SECRET_KEY=... FI_PROJECT_NAME=pi-sdk-e2e \ +PI_PROVIDER=my-gateway PI_MODEL=your-model-id MY_GATEWAY_API_KEY=... \ +./run_e2e.sh +``` + +To run the script directly against your own Pi setup (`~/.pi/agent` or `PI_CODING_AGENT_DIR`): + ```bash npm install FI_API_KEY=... FI_SECRET_KEY=... FI_PROJECT_NAME=pi-sdk node trace_session.mjs "List the files here" ``` -Content (prompts, tool arguments, tool results, answers) is off by default. Set `FI_HIDE_INPUTS=false` and/or `FI_HIDE_OUTPUTS=false` to include it. +`trace_session.mjs` exits with code 1 when the model call failed or was aborted. Content (prompts, tool arguments, tool results, answers, error text) is off by default. Set `FI_HIDE_INPUTS=false` and/or `FI_HIDE_OUTPUTS=false` to include it. diff --git a/examples/coding-agents/pi-sdk/models.example.json b/examples/coding-agents/pi-sdk/models.example.json index c85c55a71..fd971f76d 100644 --- a/examples/coding-agents/pi-sdk/models.example.json +++ b/examples/coding-agents/pi-sdk/models.example.json @@ -1,10 +1,10 @@ { "providers": { "my-gateway": { - "baseUrl": "http://127.0.0.1:8317/v1", + "baseUrl": "https://llm-gateway.example.com/v1", "api": "openai-completions", "apiKey": "$MY_GATEWAY_API_KEY", - "models": [{ "id": "gpt-6-luna" }] + "models": [{ "id": "your-model-id" }] } } } diff --git a/examples/coding-agents/pi-sdk/run_e2e.sh b/examples/coding-agents/pi-sdk/run_e2e.sh index 71e418e21..1e8a18d2b 100755 --- a/examples/coding-agents/pi-sdk/run_e2e.sh +++ b/examples/coding-agents/pi-sdk/run_e2e.sh @@ -1,21 +1,24 @@ #!/usr/bin/env bash -# End-to-end check: one real Pi SDK session, traced into a Future AGI collector. +# End-to-end run: one real Pi SDK session, traced into Future AGI. # -# FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=http://localhost:14338 \ -# PI_PROVIDER=my-gateway PI_MODEL=gpt-6-luna MY_GATEWAY_API_KEY=... \ +# FI_API_KEY=... FI_SECRET_KEY=... [FI_BASE_URL=] \ +# PI_PROVIDER=my-gateway PI_MODEL=your-model-id MY_GATEWAY_API_KEY=... \ # ./run_e2e.sh [workdir] # -# Creates an isolated Pi agent dir (models.json from $PI_MODELS_JSON, default -# models.example.json) and a throwaway project with a README.md, installs the -# example, and runs one prompt that uses the ls and read tools. The prompt -# carries a unique marker: with the default settings the marker must NOT reach -# Future AGI; rerun with FI_HIDE_INPUTS=false to see it on the run span. -# TRACEAI_PI_PACKAGE may point at a local tarball (absolute path) of -# @traceai/pi-coding-agent, for example one built with `pnpm pack`. +# Creates an isolated Pi agent dir and a throwaway project with a README.md, +# installs the example, and runs one prompt that uses the ls and read tools. +# Pi's models.json comes from $PI_MODELS_JSON, or models.example.json: edit its +# placeholder baseUrl and model id first. The prompt carries a unique marker: +# with the default settings the marker must NOT reach Future AGI; rerun with +# FI_HIDE_INPUTS=false to see it on the run span. Exits non-zero when the model +# call failed or was aborted. +# TRACEAI_PI_PACKAGE may point at a local tarball of @traceai/pi-coding-agent +# (built with `pnpm pack`) until the package is on npm. set -euo pipefail HERE="$(cd "$(dirname "$0")" && pwd)" WORK="${1:-$(mktemp -d)}" +mkdir -p "$WORK" && WORK="$(cd "$WORK" && pwd)" : "${FI_API_KEY:?set FI_API_KEY}" "${FI_SECRET_KEY:?set FI_SECRET_KEY}" : "${PI_PROVIDER:?set PI_PROVIDER}" "${PI_MODEL:?set PI_MODEL}" export FI_PROJECT_NAME="${FI_PROJECT_NAME:-pi-sdk-e2e}" @@ -24,15 +27,21 @@ mkdir -p "$WORK/app" "$WORK/agent" "$WORK/project" cp "$HERE/package.json" "$HERE/trace_session.mjs" "$WORK/app/" cp "${PI_MODELS_JSON:-$HERE/models.example.json}" "$WORK/agent/models.json" printf '# Throwaway project\n\nCreated by run_e2e.sh.\n' > "$WORK/project/README.md" +PKG="" +if [ -n "${TRACEAI_PI_PACKAGE:-}" ]; then + PKG="$(cd "$(dirname "$TRACEAI_PI_PACKAGE")" && pwd)/$(basename "$TRACEAI_PI_PACKAGE")" +fi cd "$WORK/app" -if [ -n "${TRACEAI_PI_PACKAGE:-}" ]; then - npm pkg set "dependencies.@traceai/pi-coding-agent=file:$TRACEAI_PI_PACKAGE" +if [ -n "$PKG" ]; then + npm pkg set "dependencies.@traceai/pi-coding-agent=file:$PKG" fi npm install --ignore-scripts --silent MARKER="PI-SDK-MARKER-$(date +%s)-$$" +status=0 PI_CODING_AGENT_DIR="$WORK/agent" PI_CWD="$WORK/project" \ - node trace_session.mjs "List the files in this folder, then read README.md and give me its title. $MARKER" + node trace_session.mjs "List the files in this folder, then read README.md and give me its title. $MARKER" || status=$? echo "marker=$MARKER" echo "project=$FI_PROJECT_NAME" +exit "$status" diff --git a/examples/coding-agents/pi-sdk/trace_session.mjs b/examples/coding-agents/pi-sdk/trace_session.mjs index c91b90b28..ebf101865 100644 --- a/examples/coding-agents/pi-sdk/trace_session.mjs +++ b/examples/coding-agents/pi-sdk/trace_session.mjs @@ -5,7 +5,7 @@ // Environment: // FI_API_KEY, FI_SECRET_KEY Future AGI keys (sent as X-Api-Key / X-Secret-Key) // FI_PROJECT_NAME Future AGI project (default: pi-sdk) -// FI_BASE_URL collector base URL (default: https://api.futureagi.com) +// FI_BASE_URL self-hosted Future AGI (default: https://api.futureagi.com) // FI_HIDE_INPUTS=false opt in to prompts and tool arguments // FI_HIDE_OUTPUTS=false opt in to answers and tool results // PI_CODING_AGENT_DIR Pi agent dir with models.json / auth.json (default: ~/.pi/agent) @@ -25,7 +25,8 @@ const tracerProvider = register({ const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider }); const createTracedSession = instrumentation.wrapCreateAgentSession(createAgentSession); -const options = { cwd, sessionManager: SessionManager.inMemory(cwd), tools: ["read", "ls", "bash"] }; +// Only the tools the example prompt needs; Pi has no approval step for tools. +const options = { cwd, sessionManager: SessionManager.inMemory(cwd), tools: ["read", "ls"] }; if (process.env.PI_PROVIDER && process.env.PI_MODEL) { const modelRuntime = await ModelRuntime.create(); const model = modelRuntime.getModel(process.env.PI_PROVIDER, process.env.PI_MODEL); @@ -36,8 +37,14 @@ if (process.env.PI_PROVIDER && process.env.PI_MODEL) { const { session } = await createTracedSession(options); try { await session.prompt(prompt); - console.log(session.getLastAssistantText()); + console.log(session.getLastAssistantText() ?? ""); console.error(`pi session id: ${session.sessionId}`); + // Pi resolves prompt() even when the model call failed; report it in the exit code. + const last = session.messages.findLast((message) => message.role === "assistant"); + if (last?.stopReason === "error" || last?.stopReason === "aborted") { + console.error(`pi run ended with stopReason=${last.stopReason}`); + process.exitCode = 1; + } } finally { session.dispose(); await tracerProvider.shutdown(); diff --git a/src/pages/docs/integrations/coding-agents/pi-sdk.mdx b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx index 6af37f783..77eb84c87 100644 --- a/src/pages/docs/integrations/coding-agents/pi-sdk.mdx +++ b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx @@ -1,6 +1,6 @@ --- title: "Pi Coding Agent SDK tracing with Future AGI" -description: "Trace Pi Coding Agent SDK sessions (createAgentSession) in Future AGI with @traceai/pi-coding-agent: one trace per agent run, with turns, model calls, tokens, tool calls, and errors. Prompt and code content stay off by default." +description: "Trace Pi Coding Agent SDK sessions (createAgentSession) in Future AGI with @traceai/pi-coding-agent: one trace per agent run, with turns, model calls, tokens, tool calls, compaction, and errors. Prompt, code, and error text stay off by default." --- ## About @@ -15,25 +15,31 @@ invoke_agent pi one agent run (session.prompt), session.id = Pi s ├── pi.turn │ ├── chat gpt-6-luna │ └── execute_tool read -└── pi.turn - └── chat gpt-6-luna final answer +├── pi.turn +│ └── chat gpt-6-luna final answer +└── pi.compaction only when Pi compacts the context during the run ``` This is not the same as: - **Pi, the command-line app** (`pi`). Interactive Pi sessions are traced with a Pi extension, not with this package. See the Pi page in this section. - **Oh My Pi** (`omp`, the omp.sh fork). It has its own hook API and its own page. -- **Model-provider tracing.** `@traceai/openai` or `@traceai/anthropic` trace raw provider calls. Pi makes its model calls itself; running a provider instrumentor in the same process adds duplicate model spans. +- **Model-provider tracing.** `@traceai/openai` or `@traceai/anthropic` trace provider SDK calls. Pi makes its model calls itself; a provider instrumentor in the same process can add duplicate model spans if it instruments the `openai` or `@anthropic-ai/sdk` client that Pi loads. + + +**Installed the Pi extension too? Avoid double traces.** If the Pi page's extension (`futureagi-otlp.ts`) is in `~/.pi/agent/extensions/` (or `$PI_CODING_AGENT_DIR/extensions/`) or in the project's `.pi/extensions/`, `createAgentSession()` discovers it with its default resource loader, and an SDK session that calls `session.bindExtensions()` (as `AgentSessionRuntime` apps and MCP setups do) starts it. Each prompt is then traced twice: once by this package and once by the extension, with duplicate model spans and tokens. Use one or the other for a given process, or create SDK sessions with a resource loader that leaves the extension out, as shown in [Troubleshooting](#troubleshooting). + | | | | --- | --- | | Tested with | `@earendil-works/pi-coding-agent` 1.2.0, Node 26, `@traceai/pi-coding-agent` 0.1.0 | -| Pi docs used | [SDK](https://pi.dev/docs/latest/sdk), [Models](https://pi.dev/docs/latest/models), `AgentSessionEvent` in the 1.2.0 type declarations, read 2026-10-14 | +| Pi versions | 0.80.5 and later, including 1.x: a run ends when Pi reports it settled (`agent_settled`). 0.75.4 to 0.80.3 have no `agent_settled`: a run ends at the first `agent_end` that does not announce a retry, so a run Pi continues after that point (overflow-compaction recovery, queued messages) is split into two traces, and a compaction after that `agent_end` is its own trace. Before 0.75.4, each automatic retry is also its own trace. | +| Pi docs used | [SDK](https://pi.dev/docs/latest/sdk), [Models](https://pi.dev/docs/latest/models), `AgentSessionEvent` and `DefaultResourceLoaderOptions` in the 1.2.0 type declarations, the 0.80.x changelog, read 2026-10-15 | | Requires | Node.js 22.19 or newer (Pi's own requirement) | -| Source | [`@traceai/pi-coding-agent`](https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent), [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk) | +| Source | [`@traceai/pi-coding-agent`](https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent) (until it is released: [traceAI PR #252](https://github.com/future-agi/traceAI/pull/252)), [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk) | -Pi moved from the `@mariozechner/*` npm scope to `@earendil-works/*` at version 0.74.0. Install `@earendil-works/pi-coding-agent`. The tracing package reads the session object by shape and does not depend on either scope. +Pi moved from the `@mariozechner/*` npm scope to `@earendil-works/*` at version 0.74.0. Install `@earendil-works/pi-coding-agent`. The tracing package reads the session object by shape and does not depend on either scope, but every `@mariozechner/*` release predates 0.75.4 (see Pi versions above). ## 1. Install @@ -42,6 +48,19 @@ Pi moved from the `@mariozechner/*` npm scope to `@earendil-works/*` at version npm install @traceai/pi-coding-agent @traceai/fi-core @earendil-works/pi-coding-agent ``` +`@traceai/pi-coding-agent` ships with [traceAI PR #252](https://github.com/future-agi/traceAI/pull/252). Until it is released on npm, build it from that PR and install the tarball: + +```bash +git clone https://github.com/future-agi/traceAI.git && cd traceAI +git fetch origin pull/252/head:pi-coding-agent && git checkout pi-coding-agent +cd typescript && pnpm install --frozen-lockfile && pnpm run build:deps +cd packages/traceai_pi_coding_agent && pnpm run build && pnpm pack # writes traceai-pi-coding-agent-0.1.0.tgz +# then, in your app: +npm install /path/to/traceAI/typescript/packages/traceai_pi_coding_agent/traceai-pi-coding-agent-0.1.0.tgz @traceai/fi-core @earendil-works/pi-coding-agent +``` + +Use `pnpm pack`, not `npm pack`: it turns the workspace dependencies into the published `@traceai/fi-core` versions. + ## 2. Add your Future AGI keys ```bash .env @@ -50,7 +69,7 @@ FI_SECRET_KEY=your-futureagi-secret-key FI_PROJECT_NAME=my-pi-agent ``` -`@traceai/fi-core` reads these and sends the keys as the `X-Api-Key` / `X-Secret-Key` headers. They are never written onto spans. For a self-hosted Future AGI, also set `FI_BASE_URL` to your collector, for example `http://localhost:14338`. +`@traceai/fi-core` reads these from the process environment and sends the keys as the `X-Api-Key` / `X-Secret-Key` headers. They are never written onto spans. Node does not read `.env` files by itself: export the variables in your shell, or start the app with `node --env-file=.env app.mjs`. For a [self-hosted Future AGI](/docs/self-hosting), also set `FI_BASE_URL` (traces go to `$FI_BASE_URL/tracer/v1/traces`), for example `http://localhost:4318`. ## 3. Trace your sessions @@ -80,38 +99,44 @@ If something else creates the session (for example an `AgentSessionRuntime`), tr const stop = instrumentation.instrumentSession(session); // stop() ends tracing for that session ``` -An `AgentSessionRuntime` creates a new `AgentSession` on new, resume, fork, and import. Call `instrumentSession(runtime.session)` again after each replacement, as Pi's docs require for any subscription. +A session is traced by one instrumentation at a time: a second `instrumentSession()` call for the same session returns the first `stop` function. After `stop()`, calling it again traces the session again. An `AgentSessionRuntime` creates a new `AgentSession` on new, resume, fork, and import. Call `instrumentSession(runtime.session)` again after each replacement, as Pi's docs require for any subscription. The model and credentials come from Pi as usual: `~/.pi/agent/models.json` and `auth.json`, or `PI_CODING_AGENT_DIR`. The tracing does not change which endpoint Pi calls. ## 4. Run it -A runnable copy is in [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk). `run_e2e.sh` creates an isolated Pi agent directory and a throwaway project, then runs one prompt that uses the `ls` and `read` tools: +A runnable copy is in [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk). `run_e2e.sh` creates an isolated Pi agent directory and a throwaway project, then runs one prompt that uses the `ls` and `read` tools. The session only gets the `read` and `ls` tools. + +First tell Pi which model to use: edit `models.example.json` (it defines a placeholder OpenAI-compatible gateway, `https://llm-gateway.example.com/v1`, with model `your-model-id`) to your endpoint and model id, or point `PI_MODELS_JSON` at your own Pi `models.json`. Then run it with the provider and model names from that file: ```bash FI_API_KEY=... FI_SECRET_KEY=... FI_PROJECT_NAME=pi-sdk-e2e \ -PI_PROVIDER=my-gateway PI_MODEL=gpt-6-luna MY_GATEWAY_API_KEY=... \ +PI_PROVIDER=my-gateway PI_MODEL=your-model-id MY_GATEWAY_API_KEY=... \ examples/coding-agents/pi-sdk/run_e2e.sh ``` +Set `TRACEAI_PI_PACKAGE` to the absolute path of the tarball from step 1 until the package is on npm. The script prints the answer and the Pi session id, and exits non-zero when the model call failed or was aborted. + Open the project in **Observe**. Each run is one trace; all runs of a session share the same `session.id` and group under **Sessions**. ## What is captured | Pi SDK event | Future AGI span | Fields | | --- | --- | --- | -| `agent_start` .. `agent_settled` | **invoke_agent pi** (`AGENT`) | `session.id`, `gen_ai.conversation.id`, `pi.session.name`; run totals (`pi.agent.attempts`, turn, model-call and tool-call counts, `pi.agent.usage.*`, `pi.agent.cost.total`); retry and compaction events | +| `agent_start` .. `agent_settled` (Pi before 0.80.5: the first `agent_end` that does not announce a retry) | **invoke_agent pi** (`AGENT`) | `session.id`, `gen_ai.conversation.id`; run totals: `pi.agent.attempts` (agent loops in the run: the first, each automatic retry, and each continuation), `pi.agent.retry_count`, turn, model-call and tool-call counts, `pi.agent.usage.*`, `pi.agent.cost.total`; retry events. `pi.session.name` only when you opt in to inputs | | `turn_start` .. `turn_end` | **pi.turn** (`CHAIN`) | turn index, stop reason, tool-result count | -| assistant `message_start` .. `message_end` | **chat <model>** (`LLM`) | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.provider.name`, `pi.api`, `gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens` / `cache_read_tokens` / `cache_write_tokens`, `gen_ai.cost.total` when Pi priced the call, finish reason, names of the tools it called. Start and end come from the message's own request timestamp and duration | +| assistant `message_start` .. `message_end` | **chat <model>** (`LLM`) | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.provider.name`, `pi.api`, `gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens` / `cache_read_tokens` / `cache_write_tokens`, `gen_ai.cost.total` when Pi priced the call, `gen_ai.response.finish_reasons` (Pi's stop reason, with `toolUse` reported as `tool_calls`) and `pi.stop_reason`, names of the tools it called. Start and end come from the message's own request timestamp and duration | | `tool_execution_start` .. `tool_execution_end` | **execute_tool <tool>** (`TOOL`) | `gen_ai.tool.name`, `gen_ai.tool.call.id`, duration, `pi.tool.is_error`. A tool called by another tool (`parentToolCallId`) nests under the calling tool | +| `compaction_start` .. `compaction_end` | **pi.compaction** (`CHAIN`) | `pi.compaction.reason` (`manual`, `threshold`, `overflow`), tokens before and estimated after, and the summarization call's `gen_ai.usage.*` and cost. Inside a run it is a child of the run; `session.compact()` or the check Pi runs before a prompt makes its own one-span trace in the same session | Input tokens follow the OpenTelemetry convention and include cached tokens. Pi's own `usage.input` excludes cache reads and writes, so it is kept separately as `pi.usage.uncached_input_tokens`. Run totals use `pi.agent.*` keys, not `gen_ai.usage.*`, so a trace's tokens are not counted twice. Errors: -- A model call that ends with `stopReason` `error` or `aborted` is `ERROR`, with the provider's error message as the status. Its turn is `ERROR` too. -- A run that ends aborted, or whose last model call failed, is `ERROR`. When Pi retries a failed call and the retry succeeds, the run is `OK` and the failed call keeps its own `ERROR` span. -- A tool result with `isError` is `ERROR`. +- A model call that ends with `stopReason` `error` or `aborted` is `ERROR`, and so is its turn. By default the status message holds only the stop reason, the HTTP status, and the provider's error type, for example `error: HTTP 400 invalid_request_error`, and `error.type` holds the error type. The provider's own error text is exported only when you opt in to outputs (see Privacy). +- A run that ends aborted, or whose last model call failed, is `ERROR` with the same status. When Pi retries a failed call and the retry succeeds, the run is `OK` and the failed call keeps its own `ERROR` span. +- A tool result with `isError` is `ERROR`, with the status `tool reported an error (result hidden)` unless you opt in to outputs. +- A failed compaction is `ERROR` with `compaction failed`. - Disposing the session mid-run ends the open spans with `pi.incomplete=true`. ## What is not available @@ -119,15 +144,16 @@ Errors: - **The model request payload.** Session events do not include the transcript Pi sends on each call, so model spans carry no input messages. The user prompt is on the run span when you opt in to content. - **Streaming detail.** No time to first token and no per-delta events; `message_update` deltas are not traced. - **Cost for custom models.** Models you add in `models.json` without prices report a cost of 0, and no cost attribute is written. +- **The compaction model.** `compaction_end` reports the summarization call's tokens but not which model made it, so `pi.compaction` has no model attribute. - **The system prompt.** Never exported, even with content on. - **Extension-only events.** `before_provider_request`, `before_provider_headers`, and other extension hooks are not part of `session.subscribe()` and are not used. ## Privacy -By default only structure and metadata leave your process: session id, model, provider, token counts, finish reasons, tool names and call ids, durations, and statuses. +By default only structure and metadata leave your process: session id, model, provider, token counts and cost, finish reasons, tool names and call ids, durations, counts, compaction reason and token counts, and statuses. An error status carries only the stop reason, HTTP status, and provider error type. -- **Not exported by default:** the user prompt, tool arguments, tool results (file contents, command output), assistant text, tool-call arguments, and tool error text. -- **Never exported:** the system prompt and the Pi working directory. +- **Not exported by default:** the user prompt, tool arguments, tool results (file contents, command output), assistant text, tool-call arguments, the session name, the compaction summary, and error text: the provider's error message, tool error text, and compaction error text. Some gateways echo request content or account details in their error messages, which is why they stay off. +- **Never exported:** the system prompt. Pi's working directory is never added as an attribute, but with content on, tool arguments, tool results, and error text contain absolute file paths, which usually include your username. To include content, opt in with either the instrumentation options or environment variables: @@ -136,16 +162,18 @@ new PiCodingAgentInstrumentation({ tracerProvider, traceConfig: { hideInputs: fa ``` ```bash -FI_HIDE_INPUTS=false # prompt (run span input), tool arguments -FI_HIDE_OUTPUTS=false # answers (model and run span output), tool results, tool error text +FI_HIDE_INPUTS=false # prompt (run span input), tool arguments, tool-call arguments, session name +FI_HIDE_OUTPUTS=false # answers (model and run span output), tool results, compaction summary, error text (provider errors up to 1,000 characters) ``` -Only the exact value `false` turns capture on; any other value keeps content hidden. An explicit `traceConfig` value wins over the environment. Each value is cut at 16,384 characters (`maxContentLength`). +Only `false` (any case, surrounding spaces ignored) turns capture on; any other value keeps content hidden. An explicit `traceConfig` value wins over the environment. Each value is cut at 16,384 characters (`maxContentLength`). ## Troubleshooting +**`register()` throws `FI_PROJECT_NAME is not set`.** Set `FI_PROJECT_NAME` or pass `projectName`. If you keep the variables in a `.env` file, load it (`node --env-file=.env app.mjs`). + **No trace appears.** -1. `register()` needs a project name (`FI_PROJECT_NAME` or `projectName`) and both keys. +1. Both keys must be set (`FI_API_KEY`, `FI_SECRET_KEY`), and `FI_BASE_URL` must point at your collector when you self-host. 2. A short script must `await tracerProvider.shutdown()` before it exits, or the last spans are lost. 3. The session must be traced before `prompt()`: create it through the wrapper, or call `instrumentSession()` first. @@ -153,4 +181,24 @@ Only the exact value `false` turns capture on; any other value keeps content hid **The run span is not under my request span.** The run is parented to the active span only when your app registered an OpenTelemetry context manager (for example `AsyncLocalStorageContextManager`); otherwise each run starts its own trace. -**Every model call appears twice.** A provider instrumentor (`@traceai/openai`, `@traceai/anthropic`) is also tracing Pi's HTTP calls in the same process. Remove it, or accept both. +**Every prompt appears as two traces.** One trace comes from this package (instrumentation scope `@traceai/pi-coding-agent`, root `invoke_agent pi`) and one from the Pi extension (scope `futureagi.pi`, root `pi.session`). The extension is installed in `~/.pi/agent/extensions/`, `$PI_CODING_AGENT_DIR/extensions/`, or the project's `.pi/extensions/`, and your app calls `session.bindExtensions()`. Remove the extension for this process, or give `createAgentSession()` a `DefaultResourceLoader` that leaves it out: + +```typescript +import { createAgentSession, DefaultResourceLoader, getAgentDir, SessionManager } from "@earendil-works/pi-coding-agent"; + +const resourceLoader = new DefaultResourceLoader({ + cwd: process.cwd(), + agentDir: getAgentDir(), + // Drop only the Future AGI Pi extension; `noExtensions: true` drops every discovered extension. + extensionsOverride: (base) => ({ + ...base, + extensions: base.extensions.filter((extension) => !extension.path.endsWith("futureagi-otlp.ts")), + }), +}); +await resourceLoader.reload(); +const { session } = await createTracedSession({ resourceLoader, sessionManager: SessionManager.inMemory() }); +``` + +**Every model call appears twice.** A provider instrumentor (`@traceai/openai`, `@traceai/anthropic`) in the same process can add a second model span if it instruments the `openai` or `@anthropic-ai/sdk` client that Pi loads. Remove it, or accept both. + +**Runs merge or split.** With Pi older than 0.80.5 there is no `agent_settled`, so a run ends at `agent_end`; see Pi versions at the top of the page. Upgrade Pi for exact run boundaries. From 0d13a2231418388dec0ce923d21f22a15a834fe6 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Thu, 15 Oct 2026 13:48:36 -0700 Subject: [PATCH 3/5] docs(pi-sdk): cancelled-retry run end, filtered source install, error wording - Pi versions row and the run-span row: on Pi 0.75.4 to 0.80.3 a run whose automatic retry is cancelled during its backoff now ends there with the failed call's status (traceAI#252). - Install from source uses pnpm install --filter "@traceai/pi-coding-agent...": without it pnpm installs and builds the whole traceAI TypeScript workspace (40+ packages); the filtered flow (install, build:deps, build, pnpm pack) was checked in a clean clone and takes about half the time and disk. - The span-tree sample uses your-model-id, matching the runnable example. - The default error status can also carry the error's class name (for example error: TypeError); Errors and Privacy now say so. Co-Authored-By: Claude Opus 5.5 --- .../docs/integrations/coding-agents/pi-sdk.mdx | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/pages/docs/integrations/coding-agents/pi-sdk.mdx b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx index 77eb84c87..eed731b2b 100644 --- a/src/pages/docs/integrations/coding-agents/pi-sdk.mdx +++ b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx @@ -10,13 +10,13 @@ The [Pi Coding Agent SDK](https://pi.dev/docs/latest/sdk) embeds the Pi coding a ``` invoke_agent pi one agent run (session.prompt), session.id = Pi session id ├── pi.turn one model response and the tools it called -│ ├── chat gpt-6-luna model call: model, provider, tokens, cost, finish reason +│ ├── chat your-model-id model call: model, provider, tokens, cost, finish reason │ └── execute_tool ls tool call: name, call id, duration, error flag ├── pi.turn -│ ├── chat gpt-6-luna +│ ├── chat your-model-id │ └── execute_tool read ├── pi.turn -│ └── chat gpt-6-luna final answer +│ └── chat your-model-id final answer └── pi.compaction only when Pi compacts the context during the run ``` @@ -33,7 +33,7 @@ This is not the same as: | | | | --- | --- | | Tested with | `@earendil-works/pi-coding-agent` 1.2.0, Node 26, `@traceai/pi-coding-agent` 0.1.0 | -| Pi versions | 0.80.5 and later, including 1.x: a run ends when Pi reports it settled (`agent_settled`). 0.75.4 to 0.80.3 have no `agent_settled`: a run ends at the first `agent_end` that does not announce a retry, so a run Pi continues after that point (overflow-compaction recovery, queued messages) is split into two traces, and a compaction after that `agent_end` is its own trace. Before 0.75.4, each automatic retry is also its own trace. | +| Pi versions | 0.80.5 and later, including 1.x: a run ends when Pi reports it settled (`agent_settled`). 0.75.4 to 0.80.3 have no `agent_settled`: a run ends at the first `agent_end` that does not announce a retry, so a run Pi continues after that point (overflow-compaction recovery, queued messages) is split into two traces, and a compaction after that `agent_end` is its own trace. A run whose automatic retry you cancel during its backoff (`session.abort()` or `abortRetry()`) ends at that cancellation, with the failed call's status. Before 0.75.4, each automatic retry is also its own trace. | | Pi docs used | [SDK](https://pi.dev/docs/latest/sdk), [Models](https://pi.dev/docs/latest/models), `AgentSessionEvent` and `DefaultResourceLoaderOptions` in the 1.2.0 type declarations, the 0.80.x changelog, read 2026-10-15 | | Requires | Node.js 22.19 or newer (Pi's own requirement) | | Source | [`@traceai/pi-coding-agent`](https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent) (until it is released: [traceAI PR #252](https://github.com/future-agi/traceAI/pull/252)), [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk) | @@ -53,13 +53,13 @@ npm install @traceai/pi-coding-agent @traceai/fi-core @earendil-works/pi-coding- ```bash git clone https://github.com/future-agi/traceAI.git && cd traceAI git fetch origin pull/252/head:pi-coding-agent && git checkout pi-coding-agent -cd typescript && pnpm install --frozen-lockfile && pnpm run build:deps +cd typescript && pnpm install --frozen-lockfile --filter "@traceai/pi-coding-agent..." && pnpm run build:deps cd packages/traceai_pi_coding_agent && pnpm run build && pnpm pack # writes traceai-pi-coding-agent-0.1.0.tgz # then, in your app: npm install /path/to/traceAI/typescript/packages/traceai_pi_coding_agent/traceai-pi-coding-agent-0.1.0.tgz @traceai/fi-core @earendil-works/pi-coding-agent ``` -Use `pnpm pack`, not `npm pack`: it turns the workspace dependencies into the published `@traceai/fi-core` versions. +Without `--filter`, `pnpm install` installs and builds every package in the traceAI TypeScript workspace (40+ packages). The filter selects only this package and the two `@traceai` packages it builds from, which roughly halves the install time and disk use. Use `pnpm pack`, not `npm pack`: it turns the workspace dependencies into the published `@traceai/fi-core` versions. ## 2. Add your Future AGI keys @@ -123,7 +123,7 @@ Open the project in **Observe**. Each run is one trace; all runs of a session sh | Pi SDK event | Future AGI span | Fields | | --- | --- | --- | -| `agent_start` .. `agent_settled` (Pi before 0.80.5: the first `agent_end` that does not announce a retry) | **invoke_agent pi** (`AGENT`) | `session.id`, `gen_ai.conversation.id`; run totals: `pi.agent.attempts` (agent loops in the run: the first, each automatic retry, and each continuation), `pi.agent.retry_count`, turn, model-call and tool-call counts, `pi.agent.usage.*`, `pi.agent.cost.total`; retry events. `pi.session.name` only when you opt in to inputs | +| `agent_start` .. `agent_settled` (Pi before 0.80.5: the first `agent_end` that does not announce a retry, or a cancelled retry) | **invoke_agent pi** (`AGENT`) | `session.id`, `gen_ai.conversation.id`; run totals: `pi.agent.attempts` (agent loops in the run: the first, each automatic retry, and each continuation), `pi.agent.retry_count`, turn, model-call and tool-call counts, `pi.agent.usage.*`, `pi.agent.cost.total`; retry events. `pi.session.name` only when you opt in to inputs | | `turn_start` .. `turn_end` | **pi.turn** (`CHAIN`) | turn index, stop reason, tool-result count | | assistant `message_start` .. `message_end` | **chat <model>** (`LLM`) | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.provider.name`, `pi.api`, `gen_ai.usage.input_tokens` / `output_tokens` / `total_tokens` / `cache_read_tokens` / `cache_write_tokens`, `gen_ai.cost.total` when Pi priced the call, `gen_ai.response.finish_reasons` (Pi's stop reason, with `toolUse` reported as `tool_calls`) and `pi.stop_reason`, names of the tools it called. Start and end come from the message's own request timestamp and duration | | `tool_execution_start` .. `tool_execution_end` | **execute_tool <tool>** (`TOOL`) | `gen_ai.tool.name`, `gen_ai.tool.call.id`, duration, `pi.tool.is_error`. A tool called by another tool (`parentToolCallId`) nests under the calling tool | @@ -133,7 +133,7 @@ Input tokens follow the OpenTelemetry convention and include cached tokens. Pi's Errors: -- A model call that ends with `stopReason` `error` or `aborted` is `ERROR`, and so is its turn. By default the status message holds only the stop reason, the HTTP status, and the provider's error type, for example `error: HTTP 400 invalid_request_error`, and `error.type` holds the error type. The provider's own error text is exported only when you opt in to outputs (see Privacy). +- A model call that ends with `stopReason` `error` or `aborted` is `ERROR`, and so is its turn. By default the status message holds only the stop reason, the HTTP status, and the provider's error type or, when there is none, the error's class name, for example `error: HTTP 400 invalid_request_error` or `error: TypeError`, and `error.type` holds that type or class name. The provider's own error text is exported only when you opt in to outputs (see Privacy). - A run that ends aborted, or whose last model call failed, is `ERROR` with the same status. When Pi retries a failed call and the retry succeeds, the run is `OK` and the failed call keeps its own `ERROR` span. - A tool result with `isError` is `ERROR`, with the status `tool reported an error (result hidden)` unless you opt in to outputs. - A failed compaction is `ERROR` with `compaction failed`. @@ -150,7 +150,7 @@ Errors: ## Privacy -By default only structure and metadata leave your process: session id, model, provider, token counts and cost, finish reasons, tool names and call ids, durations, counts, compaction reason and token counts, and statuses. An error status carries only the stop reason, HTTP status, and provider error type. +By default only structure and metadata leave your process: session id, model, provider, token counts and cost, finish reasons, tool names and call ids, durations, counts, compaction reason and token counts, and statuses. An error status carries only the stop reason, HTTP status, and provider error type or error class name. - **Not exported by default:** the user prompt, tool arguments, tool results (file contents, command output), assistant text, tool-call arguments, the session name, the compaction summary, and error text: the provider's error message, tool error text, and compaction error text. Some gateways echo request content or account details in their error messages, which is why they stay off. - **Never exported:** the system prompt. Pi's working directory is never added as an attribute, but with content on, tool arguments, tool results, and error text contain absolute file paths, which usually include your username. From 07aaf4cba462a54000e4d7eeb8c67a9048e4f2e7 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Wed, 21 Oct 2026 15:14:26 -0700 Subject: [PATCH 4/5] docs(pi-sdk): drop the build-from-PR install steps; the page goes live after @traceai/pi-coding-agent is on npm (TH-8264) Co-Authored-By: Claude Opus 5.5 --- examples/coding-agents/pi-sdk/README.md | 2 +- examples/coding-agents/pi-sdk/run_e2e.sh | 2 +- .../docs/integrations/coding-agents/pi-sdk.mdx | 17 ++--------------- 3 files changed, 4 insertions(+), 17 deletions(-) diff --git a/examples/coding-agents/pi-sdk/README.md b/examples/coding-agents/pi-sdk/README.md index 35e9c262d..9ac979db5 100644 --- a/examples/coding-agents/pi-sdk/README.md +++ b/examples/coding-agents/pi-sdk/README.md @@ -9,7 +9,7 @@ Runnable example for [Pi Coding Agent SDK tracing](https://docs.futureagi.com/do | `run_e2e.sh` | Isolated end-to-end run with a throwaway project and a privacy marker | | `package.json` | Pinned to the tested Pi SDK (1.2.0) | -`@traceai/pi-coding-agent` ships with [traceAI PR #252](https://github.com/future-agi/traceAI/pull/252). Until it is on npm, build its tarball as the [docs page](https://docs.futureagi.com/docs/integrations/coding-agents/pi-sdk) shows (`pnpm pack` in a traceAI checkout of that PR) and pass it to `run_e2e.sh` as `TRACEAI_PI_PACKAGE=/abs/path/traceai-pi-coding-agent-0.1.0.tgz`, or `npm install` it here before the other dependencies. +`run_e2e.sh` installs `@traceai/pi-coding-agent` from npm. To test a local build instead, pass its `pnpm pack` tarball as `TRACEAI_PI_PACKAGE=/abs/path/traceai-pi-coding-agent-0.1.0.tgz`. Edit `models.example.json` (gateway `baseUrl` and model `id`) or point `PI_MODELS_JSON` at your own Pi `models.json`, then: diff --git a/examples/coding-agents/pi-sdk/run_e2e.sh b/examples/coding-agents/pi-sdk/run_e2e.sh index 1e8a18d2b..b0025e678 100755 --- a/examples/coding-agents/pi-sdk/run_e2e.sh +++ b/examples/coding-agents/pi-sdk/run_e2e.sh @@ -13,7 +13,7 @@ # FI_HIDE_INPUTS=false to see it on the run span. Exits non-zero when the model # call failed or was aborted. # TRACEAI_PI_PACKAGE may point at a local tarball of @traceai/pi-coding-agent -# (built with `pnpm pack`) until the package is on npm. +# (built with `pnpm pack`) to test a build that is not on npm. set -euo pipefail HERE="$(cd "$(dirname "$0")" && pwd)" diff --git a/src/pages/docs/integrations/coding-agents/pi-sdk.mdx b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx index eed731b2b..12a8ecfb8 100644 --- a/src/pages/docs/integrations/coding-agents/pi-sdk.mdx +++ b/src/pages/docs/integrations/coding-agents/pi-sdk.mdx @@ -36,7 +36,7 @@ This is not the same as: | Pi versions | 0.80.5 and later, including 1.x: a run ends when Pi reports it settled (`agent_settled`). 0.75.4 to 0.80.3 have no `agent_settled`: a run ends at the first `agent_end` that does not announce a retry, so a run Pi continues after that point (overflow-compaction recovery, queued messages) is split into two traces, and a compaction after that `agent_end` is its own trace. A run whose automatic retry you cancel during its backoff (`session.abort()` or `abortRetry()`) ends at that cancellation, with the failed call's status. Before 0.75.4, each automatic retry is also its own trace. | | Pi docs used | [SDK](https://pi.dev/docs/latest/sdk), [Models](https://pi.dev/docs/latest/models), `AgentSessionEvent` and `DefaultResourceLoaderOptions` in the 1.2.0 type declarations, the 0.80.x changelog, read 2026-10-15 | | Requires | Node.js 22.19 or newer (Pi's own requirement) | -| Source | [`@traceai/pi-coding-agent`](https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent) (until it is released: [traceAI PR #252](https://github.com/future-agi/traceAI/pull/252)), [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk) | +| Source | [`@traceai/pi-coding-agent`](https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent) in traceAI ([npm](https://www.npmjs.com/package/@traceai/pi-coding-agent)), [examples/coding-agents/pi-sdk](https://github.com/future-agi/docs/tree/main/examples/coding-agents/pi-sdk) | Pi moved from the `@mariozechner/*` npm scope to `@earendil-works/*` at version 0.74.0. Install `@earendil-works/pi-coding-agent`. The tracing package reads the session object by shape and does not depend on either scope, but every `@mariozechner/*` release predates 0.75.4 (see Pi versions above). @@ -48,19 +48,6 @@ Pi moved from the `@mariozechner/*` npm scope to `@earendil-works/*` at version npm install @traceai/pi-coding-agent @traceai/fi-core @earendil-works/pi-coding-agent ``` -`@traceai/pi-coding-agent` ships with [traceAI PR #252](https://github.com/future-agi/traceAI/pull/252). Until it is released on npm, build it from that PR and install the tarball: - -```bash -git clone https://github.com/future-agi/traceAI.git && cd traceAI -git fetch origin pull/252/head:pi-coding-agent && git checkout pi-coding-agent -cd typescript && pnpm install --frozen-lockfile --filter "@traceai/pi-coding-agent..." && pnpm run build:deps -cd packages/traceai_pi_coding_agent && pnpm run build && pnpm pack # writes traceai-pi-coding-agent-0.1.0.tgz -# then, in your app: -npm install /path/to/traceAI/typescript/packages/traceai_pi_coding_agent/traceai-pi-coding-agent-0.1.0.tgz @traceai/fi-core @earendil-works/pi-coding-agent -``` - -Without `--filter`, `pnpm install` installs and builds every package in the traceAI TypeScript workspace (40+ packages). The filter selects only this package and the two `@traceai` packages it builds from, which roughly halves the install time and disk use. Use `pnpm pack`, not `npm pack`: it turns the workspace dependencies into the published `@traceai/fi-core` versions. - ## 2. Add your Future AGI keys ```bash .env @@ -115,7 +102,7 @@ PI_PROVIDER=my-gateway PI_MODEL=your-model-id MY_GATEWAY_API_KEY=... \ examples/coding-agents/pi-sdk/run_e2e.sh ``` -Set `TRACEAI_PI_PACKAGE` to the absolute path of the tarball from step 1 until the package is on npm. The script prints the answer and the Pi session id, and exits non-zero when the model call failed or was aborted. +The script prints the answer and the Pi session id, and exits non-zero when the model call failed or was aborted. Open the project in **Observe**. Each run is one trace; all runs of a session share the same `session.id` and group under **Sessions**. From 2adbd9819ea4458341de48ba20f63149c4864a7a Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Thu, 5 Nov 2026 16:35:04 -0800 Subject: [PATCH 5/5] docs(ci): pin browser build to supported Ubuntu runner Playwright 1.56.1 cannot install Chromium dependencies on the Ubuntu 26.04 image now used by ubuntu-latest. Keep the build and browser checks enabled on Ubuntu 24.04. Co-Authored-By: Claude Code --- .github/workflows/pr-checks.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 2bb5470c3..5090bd917 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -76,7 +76,8 @@ jobs: docs-build: name: Docs build & Pagefind - runs-on: ubuntu-latest + # Playwright 1.56.1 supports Ubuntu 24.04, not the ubuntu-latest 26.04 image. + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4