From 3b3e4a43f0818c724b26411bdb9112838691ec2f Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Wed, 14 Oct 2026 18:02:21 -0700 Subject: [PATCH 1/7] feat(pi-coding-agent): Pi Coding Agent SDK instrumentor (TH-8264) @traceai/pi-coding-agent traces @earendil-works/pi-coding-agent sessions through AgentSession.subscribe(): one AGENT span per agent run (agent_start..agent_settled, across Pi's automatic retries), a CHAIN span per turn, an LLM span per assistant message (model, provider, tokens with cache reads added back, cost when priced, finish reason, ERROR on error/aborted) and a TOOL span per tool execution (nested by parentToolCallId, ERROR on isError). session.id is the Pi session id on every span. The SDK is ESM-only, so apps wrap createAgentSession or call instrumentSession(session). Prompts, tool arguments, tool results and answers are hidden by default (FI_HIDE_INPUTS/FI_HIDE_OUTPUTS=false or traceConfig opt in); the system prompt is never exported. No runtime or lockfile dependency on the SDK: sessions are read structurally, and contract/sdk_types.ts checks the real 1.2.0 types fit. Co-Authored-By: Claude Opus 5.5 --- .../traceai_pi_coding_agent/README.md | 115 ++++ .../contract/README.md | 21 + .../contract/real_sdk_session.mjs | 39 ++ .../contract/sdk_types.ts | 13 + .../examples/basic.mjs | 28 + .../traceai_pi_coding_agent/jest.config.js | 22 + .../traceai_pi_coding_agent/package.json | 72 +++ .../src/__tests__/fixtures/events.ts | 229 ++++++++ .../src/__tests__/helpers.ts | 78 +++ .../src/__tests__/package.test.ts | 61 +++ .../src/__tests__/sessionTracer.test.ts | 369 +++++++++++++ .../traceai_pi_coding_agent/src/attributes.ts | 103 ++++ .../traceai_pi_coding_agent/src/content.ts | 72 +++ .../traceai_pi_coding_agent/src/index.ts | 21 + .../src/instrumentation.ts | 244 +++++++++ .../src/sessionTracer.ts | 509 ++++++++++++++++++ .../traceai_pi_coding_agent/src/types.ts | 74 +++ .../traceai_pi_coding_agent/src/version.ts | 1 + .../traceai_pi_coding_agent/tsconfig.esm.json | 17 + .../tsconfig.esnext.json | 17 + .../traceai_pi_coding_agent/tsconfig.json | 14 + typescript/pnpm-lock.yaml | 37 ++ 22 files changed, 2156 insertions(+) create mode 100644 typescript/packages/traceai_pi_coding_agent/README.md create mode 100644 typescript/packages/traceai_pi_coding_agent/contract/README.md create mode 100644 typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs create mode 100644 typescript/packages/traceai_pi_coding_agent/contract/sdk_types.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/examples/basic.mjs create mode 100644 typescript/packages/traceai_pi_coding_agent/jest.config.js create mode 100644 typescript/packages/traceai_pi_coding_agent/package.json create mode 100644 typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/__tests__/helpers.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/__tests__/package.test.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/attributes.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/content.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/index.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/types.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/src/version.ts create mode 100644 typescript/packages/traceai_pi_coding_agent/tsconfig.esm.json create mode 100644 typescript/packages/traceai_pi_coding_agent/tsconfig.esnext.json create mode 100644 typescript/packages/traceai_pi_coding_agent/tsconfig.json diff --git a/typescript/packages/traceai_pi_coding_agent/README.md b/typescript/packages/traceai_pi_coding_agent/README.md new file mode 100644 index 000000000..e0a70fbb6 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/README.md @@ -0,0 +1,115 @@ +# @traceai/pi-coding-agent + +OpenTelemetry tracing for the [Pi Coding Agent SDK](https://pi.dev/docs/latest/sdk) (`createAgentSession()` / `AgentSession`), exported to Future AGI through `@traceai/fi-core`. + +Alpha. Tested against `@earendil-works/pi-coding-agent` 1.2.0 with recorded event sequences and with a real SDK session (OpenAI-compatible endpoint, built-in `ls` / `read` tools) exported to a local Future AGI collector. + +This is the in-process SDK integration. It is not the Pi command-line app (`pi`, traced with a Pi extension) and not the Oh My Pi fork (`omp`). Pi's own model calls are not instrumented a second time: do not also run a provider instrumentor (`@traceai/openai`, `@traceai/anthropic`) on the same process unless you want duplicate model spans. + +| Item | Value | +|---|---| +| Tested SDK | `@earendil-works/pi-coding-agent` 1.2.0 (Node 22.19+) | +| Package scope | Pi moved from `@mariozechner/*` to `@earendil-works/*` at 0.74.0. Sessions from either scope work: the package reads the session object structurally and has no dependency on the SDK. | +| SDK surface used | `AgentSession.sessionId`, `AgentSession.subscribe()`, `AgentSession.dispose()` | + +## Install + +```bash +npm install @traceai/pi-coding-agent @traceai/fi-core @earendil-works/pi-coding-agent +``` + +## Use + +The SDK is ESM-only and its module namespace cannot be patched. Wrap `createAgentSession`, or pass each session to `instrumentSession()` before you prompt it: + +```typescript +import { register, ProjectType } from "@traceai/fi-core"; +import { createAgentSession, SessionManager } from "@earendil-works/pi-coding-agent"; +import { PiCodingAgentInstrumentation } from "@traceai/pi-coding-agent"; + +const tracerProvider = register({ + projectType: ProjectType.OBSERVE, + projectName: "my-pi-agent", // or FI_PROJECT_NAME +}); +const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider }); + +const { session } = await instrumentation.wrapCreateAgentSession(createAgentSession)({ + sessionManager: SessionManager.inMemory(), +}); +// or: const { session } = await createAgentSession(...); instrumentation.instrumentSession(session); + +try { + await session.prompt("List the files in this folder and read README.md"); +} finally { + session.dispose(); // ends any span still open + await tracerProvider.shutdown(); // flush before a short script exits +} +``` + +`FI_API_KEY` and `FI_SECRET_KEY` are read by fi-core and sent as the `X-Api-Key` / `X-Secret-Key` headers. They are never copied onto spans. + +`instrumentSession()` returns a function that stops tracing that session. Calling it twice for one session is a no-op. An `AgentSessionRuntime` replaces its `AgentSession` on new / resume / fork / import; call `instrumentSession(runtime.session)` again after each replacement, as Pi's own docs say for any subscription. + +## Spans + +One trace per agent run (one `session.prompt()`, including Pi's automatic retries and compaction). Every span carries `session.id` = `session.sessionId`, so the runs of one session group together in Future AGI Sessions. + +| Span name | `fi.span.kind` / `gen_ai.span.kind` | From | Parent | +|---|---|---|---| +| `invoke_agent pi` | `AGENT` | `agent_start` .. `agent_settled` | the active span at `agent_start` when the app registered a context manager (for example `AsyncLocalStorageContextManager`), else none (a new trace) | +| `pi.turn` | `CHAIN` | `turn_start` .. `turn_end` | run | +| `chat ` | `LLM` | assistant `message_start` .. `message_end` (start/end from the message's `timestamp` + `durationMs`) | turn | +| `execute_tool ` | `TOOL` | `tool_execution_start` .. `tool_execution_end` | turn, or the calling tool when the event has `parentToolCallId` | + +| Field | Attributes | Source | +|---|---|---| +| Session | `session.id`, `pi.session.id`, `gen_ai.conversation.id`, `pi.session.name` | `session.sessionId`, `session.sessionName` | +| Model | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.provider.name`, `pi.api`, `pi.thinking_level` | `AssistantMessage` | +| Tokens | `gen_ai.usage.input_tokens` (= Pi `input` + `cacheRead` + `cacheWrite`), `output_tokens`, `total_tokens`, `cache_read_tokens`, `cache_write_tokens`, `output_tokens.reasoning`; `pi.usage.uncached_input_tokens` (Pi's own `input`) | `AssistantMessage.usage`, LLM spans only | +| Cost | `gen_ai.cost.total` | `usage.cost.total`, only when Pi priced the call (> 0). Models from `models.json` without prices report 0 and get no cost key | +| Finish | `gen_ai.response.finish_reasons`, `pi.stop_reason`, `pi.llm.tool_call_names`, `pi.llm.tool_call_count` | `stopReason`, `toolCall` parts | +| Tools | `gen_ai.tool.name`, `tool.name`, `gen_ai.tool.call.id`, `tool_call.id`, `pi.tool.duration_ms`, `pi.tool.is_error`, `pi.tool.parent_call_id` | tool execution events | +| Run totals | `pi.agent.attempts`, `turn_count`, `llm_call_count`, `tool_call_count`, `pi.agent.usage.*`, `pi.agent.cost.total` | summed LLM spans. Not the promoted `gen_ai.usage.*` keys, so tokens are not counted twice | +| Retries / compaction | span events `pi.auto_retry.start` / `.end`, `pi.compaction.start` / `.end` on the run span | session events | + +Errors: + +- An assistant message with `stopReason` `error` or `aborted` makes its LLM span and turn ERROR, with the provider's `errorMessage` as the status message (truncated to 1000 characters). `aborted` also sets `pi.aborted=true`. +- The run span is ERROR when the run ends aborted (`agent_settled.aborted`) or its last model call failed. A failed call that Pi retried successfully leaves the run OK; the failed call's own span stays ERROR. +- A tool result with `isError: true` makes the tool span ERROR. The status message is the tool's error text only when outputs are on; otherwise "tool reported an error (result hidden)". +- `session.dispose()` or the stop function during a run ends every open span with `pi.incomplete=true`. + +Not available from the SDK's session events, so not recorded: the provider request payload (the full transcript sent on each model call), streaming deltas and time to first token, per-request HTTP status, and the system prompt. + +## Privacy: content is off by default + +Prompts, tool arguments, tool results and assistant text of a coding agent contain source code, file contents and command output. They are **not** recorded unless you opt in: + +```typescript +new PiCodingAgentInstrumentation({ tracerProvider, traceConfig: { hideInputs: false, hideOutputs: false } }); +// or: FI_HIDE_INPUTS=false FI_HIDE_OUTPUTS=false +``` + +| Setting | Adds | +|---|---| +| `hideInputs: false` | the user prompt (`input.value` on the run span), tool arguments (`input.value` on tool spans), tool-call arguments inside the LLM output | +| `hideOutputs: false` | assistant text (`output.value` on LLM and run spans), tool results (`output.value` on tool spans), tool error text in the status | + +Precedence: `traceConfig` option, then the env var, then hidden. The env vars fail closed: only `false` (trimmed, any case) turns capture on. Values are cut at `maxContentLength` characters (default 16384). The system prompt is never exported. + +## Flushing + +`tracerProvider.shutdown()` (or this package's `shutdown()`, which calls `forceFlush()` on the providers it knows) before a short script exits. A tracing error never fails the session: every event handler catches and logs through the OpenTelemetry diag logger. + +## Troubleshooting + +- **No traces.** `register()` needs `projectName` or `FI_PROJECT_NAME`, and `FI_API_KEY` / `FI_SECRET_KEY`. +- **Nothing from a short script.** Await `tracerProvider.shutdown()` before exit. +- **Spans stop after `/new`, `/resume` or a fork.** The runtime made a new `AgentSession`; instrument it. +- **Duplicate model spans.** A provider instrumentor is also tracing Pi's HTTP calls. + +## Tests + +```bash +pnpm --filter @traceai/pi-coding-agent test +``` diff --git a/typescript/packages/traceai_pi_coding_agent/contract/README.md b/typescript/packages/traceai_pi_coding_agent/contract/README.md new file mode 100644 index 000000000..bca5eddca --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/contract/README.md @@ -0,0 +1,21 @@ +# Real-SDK contract checks + +Not part of `pnpm test` (they need the Pi SDK, a model endpoint and a collector). + +`sdk_types.ts` fails to compile if `@earendil-works/pi-coding-agent`'s +`AgentSession` / `createAgentSession` stop fitting the structural types this +package accepts. `real_sdk_session.mjs` runs one real session through the +instrumentation and exports it with `@traceai/fi-core`. + +```bash +mkdir /tmp/pi-contract && cd /tmp/pi-contract && npm init -y && npm pkg set type=module +npm install @earendil-works/pi-coding-agent@1.2.0 typescript@5 \ + /fi-core.tgz /fi-semantic-conventions.tgz /traceai-pi-coding-agent.tgz # pnpm pack each +cp /sdk_types.ts . && npx tsc --noEmit --strict --skipLibCheck \ + --module NodeNext --moduleResolution NodeNext --target ES2022 sdk_types.ts + +# models.json in $PI_CODING_AGENT_DIR defines / +FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=http://localhost:14338 FI_PROJECT_NAME=pi-sdk-contract \ +PI_CODING_AGENT_DIR=/tmp/pi-agent PI_E2E_CWD=/tmp/pi-project \ + node /real_sdk_session.mjs "List the files here and read README.md" +``` diff --git a/typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs b/typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs new file mode 100644 index 000000000..f0fe41205 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs @@ -0,0 +1,39 @@ +// One real Pi SDK session traced by @traceai/pi-coding-agent and exported to +// Future AGI. Configuration comes from the environment: +// FI_API_KEY, FI_SECRET_KEY, FI_BASE_URL, FI_PROJECT_NAME (Future AGI) +// PI_CODING_AGENT_DIR (agentDir with models.json), PI_E2E_CWD (project dir) +// argv: +import { join } from "node:path"; +import { ProjectType, register } from "@traceai/fi-core"; +import { PiCodingAgentInstrumentation } from "@traceai/pi-coding-agent"; +import { createAgentSession, ModelRuntime, SessionManager, SettingsManager } from "@earendil-works/pi-coding-agent"; + +const [provider, modelId, prompt] = process.argv.slice(2); +const agentDir = process.env.PI_CODING_AGENT_DIR; +const cwd = process.env.PI_E2E_CWD; + +const tracerProvider = register({ projectName: process.env.FI_PROJECT_NAME, projectType: ProjectType.OBSERVE }); +const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider }); +const tracedCreateAgentSession = instrumentation.wrapCreateAgentSession(createAgentSession); + +const modelRuntime = await ModelRuntime.create({ authPath: join(agentDir, "auth.json"), modelsPath: join(agentDir, "models.json") }); +const model = modelRuntime.getModel(provider, modelId); +if (!model) throw new Error(`model ${provider}/${modelId} not in models.json`); + +const { session } = await tracedCreateAgentSession({ + cwd, + agentDir, + model, + modelRuntime, + thinkingLevel: "off", + sessionManager: SessionManager.inMemory(cwd), + settingsManager: SettingsManager.inMemory({}), + tools: ["read", "bash", "ls"], +}); +try { + await session.prompt(prompt); + console.log(JSON.stringify({ sessionId: session.sessionId, answer: session.getLastAssistantText()?.slice(0, 200) })); +} finally { + session.dispose(); + await tracerProvider.shutdown(); +} diff --git a/typescript/packages/traceai_pi_coding_agent/contract/sdk_types.ts b/typescript/packages/traceai_pi_coding_agent/contract/sdk_types.ts new file mode 100644 index 000000000..7ee14d0f0 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/contract/sdk_types.ts @@ -0,0 +1,13 @@ +// Compile-time proof that the real SDK types fit the structural types +// @traceai/pi-coding-agent accepts. `tsc --noEmit` fails if they drift. +import { createAgentSession, type AgentSession, type AgentSessionEvent } from "@earendil-works/pi-coding-agent"; +import { PiCodingAgentInstrumentation, type PiSessionEvent, type PiSessionLike } from "@traceai/pi-coding-agent"; + +declare const session: AgentSession; +const asLike: PiSessionLike = session; +const instrumentation = new PiCodingAgentInstrumentation(); +const traced: typeof createAgentSession = instrumentation.wrapCreateAgentSession(createAgentSession); +const stop: () => void = instrumentation.instrumentSession(session); +declare const event: AgentSessionEvent; +const asEvent: PiSessionEvent = event as PiSessionEvent; +void [asLike, traced, stop, asEvent]; diff --git a/typescript/packages/traceai_pi_coding_agent/examples/basic.mjs b/typescript/packages/traceai_pi_coding_agent/examples/basic.mjs new file mode 100644 index 000000000..6da5a8bd9 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/examples/basic.mjs @@ -0,0 +1,28 @@ +// Trace one Pi Coding Agent SDK session in Future AGI. +// +// npm install @traceai/pi-coding-agent @traceai/fi-core @earendil-works/pi-coding-agent +// FI_API_KEY=... FI_SECRET_KEY=... FI_PROJECT_NAME=my-pi-agent node basic.mjs +// +// Pi uses the model and credentials from ~/.pi/agent (or PI_CODING_AGENT_DIR). +import { ProjectType, register } from "@traceai/fi-core"; +import { createAgentSession, SessionManager } from "@earendil-works/pi-coding-agent"; +import { PiCodingAgentInstrumentation } from "@traceai/pi-coding-agent"; + +const tracerProvider = register({ projectType: ProjectType.OBSERVE }); +const instrumentation = new PiCodingAgentInstrumentation({ + tracerProvider, + // Content is hidden by default. Opt in to prompts, tool arguments and results: + // traceConfig: { hideInputs: false, hideOutputs: false }, +}); + +const { session } = await instrumentation.wrapCreateAgentSession(createAgentSession)({ + sessionManager: SessionManager.inMemory(), +}); + +try { + await session.prompt("List the files in this folder and read README.md"); + console.log(session.getLastAssistantText()); +} finally { + session.dispose(); + await tracerProvider.shutdown(); +} diff --git a/typescript/packages/traceai_pi_coding_agent/jest.config.js b/typescript/packages/traceai_pi_coding_agent/jest.config.js new file mode 100644 index 000000000..1a1a13998 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/jest.config.js @@ -0,0 +1,22 @@ +module.exports = { + preset: 'ts-jest', + testEnvironment: 'node', + roots: ['/src'], + testMatch: ['**/__tests__/**/*.test.ts'], + testPathIgnorePatterns: ['/node_modules/'], + collectCoverageFrom: ['src/**/*.ts', '!src/**/*.d.ts', '!src/index.ts', '!src/__tests__/**'], + coverageDirectory: 'coverage', + coverageReporters: ['text', 'lcov'], + transform: { + '^.+\\.ts$': ['ts-jest', { + tsconfig: { + module: 'commonjs' + } + }] + }, + moduleNameMapper: { + '^@traceai/fi-core$': '/../fi-core/src', + '^@traceai/fi-semantic-conventions$': '/../fi-semantic-conventions/src' + }, + testTimeout: 15000 +}; diff --git a/typescript/packages/traceai_pi_coding_agent/package.json b/typescript/packages/traceai_pi_coding_agent/package.json new file mode 100644 index 000000000..969751def --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/package.json @@ -0,0 +1,72 @@ +{ + "name": "@traceai/pi-coding-agent", + "version": "0.1.0", + "description": "OpenTelemetry tracing for Pi Coding Agent SDK sessions (createAgentSession / session.subscribe)", + "main": "./dist/src/index.js", + "module": "./dist/esm/index.js", + "esnext": "./dist/esnext/index.js", + "types": "./dist/src/index.d.ts", + "exports": { + ".": { + "types": "./dist/src/index.d.ts", + "import": "./dist/esm/index.js", + "require": "./dist/src/index.js" + } + }, + "scripts": { + "prebuild": "rimraf dist", + "build": "tsc --build tsconfig.json tsconfig.esm.json tsconfig.esnext.json && tsc-alias -p tsconfig.esm.json && pnpm run postbuild", + "postbuild": "echo '{\"type\": \"module\"}' > ./dist/esm/package.json", + "test": "jest", + "prepare": "pnpm run build" + }, + "keywords": [ + "opentelemetry", + "tracing", + "instrumentation", + "pi", + "pi-coding-agent", + "coding-agent", + "agents", + "llm", + "observability" + ], + "author": "Future AGI ", + "license": "Apache-2.0", + "repository": { + "type": "git", + "url": "git+https://github.com/future-agi/traceAI.git", + "directory": "typescript/packages/traceai_pi_coding_agent" + }, + "bugs": { + "url": "https://github.com/future-agi/traceAI/issues" + }, + "homepage": "https://github.com/future-agi/traceAI/tree/main/typescript/packages/traceai_pi_coding_agent#readme", + "engines": { + "node": ">=18.0.0" + }, + "dependencies": { + "@opentelemetry/api": "^1.9.0", + "@traceai/fi-core": "workspace:^", + "@traceai/fi-semantic-conventions": "workspace:^" + }, + "devDependencies": { + "@opentelemetry/sdk-trace-base": "^2.0.1", + "@types/jest": "^29.5.14", + "@types/node": "^22.10.10", + "jest": "^29.7.0", + "rimraf": "^6.0.1", + "ts-jest": "^29.2.5", + "tsc-alias": "^1.8.10", + "typescript": "^5.7.3" + }, + "files": [ + "dist", + "src", + "!**/__tests__/**", + "!**/*.tsbuildinfo" + ], + "publishConfig": { + "access": "public" + } +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts new file mode 100644 index 000000000..1a9677e15 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts @@ -0,0 +1,229 @@ +/** + * Event sequences in the order `@earendil-works/pi-coding-agent` 1.2.0 + * emits them from `AgentSession.subscribe()`. + * + * `happyPath` mirrors a recorded real run (gpt-6-luna through an + * OpenAI-compatible endpoint, prompt "list the files, read README.md"): same + * event types, order, roles, message keys, stop reasons and usage shape. Text, + * ids and token numbers are fixtures; the *_SECRET strings let tests prove + * which content leaves the process. + */ +import { PiSessionEvent, PiSessionEventListener, PiSessionLike } from "../../types"; + +export const SESSION_ID = "0199f1d2-7c1e-7000-8000-00000000c0de"; +export const PROMPT_SECRET = "PROMPT_SECRET_list_files_please"; +export const SYSTEM_SECRET = "SYSTEM_SECRET_you_are_pi"; +export const ARG_SECRET = "ARG_SECRET_/tmp/project"; +export const RESULT_SECRET = "RESULT_SECRET_README.md_hello.js"; +export const OUTPUT_SECRET = "OUTPUT_SECRET_the_title_is_Throwaway"; +export const TOOL_ERROR_SECRET = "TOOL_ERROR_SECRET_ENOENT_missing.txt"; + +const T0 = 1_760_000_000_000; + +function usage(input: number, output: number, cacheRead = 0, cacheWrite = 0, cost = 0) { + return { + input, + output, + cacheRead, + cacheWrite, + totalTokens: input + output + cacheRead + cacheWrite, + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: cost }, + }; +} + +function assistant( + timestamp: number, + content: unknown[], + stopReason: string, + extra: Record = {}, +): Record { + return { + role: "assistant", + content, + api: "openai-completions", + provider: "cliproxy", + model: "gpt-6-luna", + responseId: `chatcmpl-${timestamp}`, + timestamp, + usage: usage(0, 0), + stopReason: "pending", + ...extra, + ...(stopReason === "pending" ? {} : { stopReason }), + }; +} + +function toolTurn( + start: number, + callId: string, + toolName: string, + args: Record, + resultText: string, + tokens: ReturnType, + isError = false, +): PiSessionEvent[] { + const call = [{ type: "toolCall", id: callId, name: toolName, arguments: args }]; + const streaming = assistant(start, call, "pending"); + const done = assistant(start, call, "toolUse", { usage: tokens, durationMs: 900, thinkingLevel: "off", rawStopReason: "tool_calls" }); + const toolResult = { + role: "toolResult", + toolCallId: callId, + toolName, + content: [{ type: "text", text: resultText }], + isError, + durationMs: 20, + timestamp: start + 1_000, + }; + return [ + { type: "turn_start" }, + { type: "message_start", message: streaming }, + { type: "message_update", message: streaming, assistantMessageEvent: { type: "toolcall_delta", delta: "{" } }, + { type: "message_end", message: done }, + { type: "tool_execution_start", toolCallId: callId, toolName, args }, + { type: "tool_execution_update", toolCallId: callId, toolName, args, partialResult: { content: [] } }, + { type: "tool_execution_end", toolCallId: callId, toolName, result: { content: [{ type: "text", text: resultText }] }, isError, durationMs: 20 }, + { type: "message_start", message: toolResult }, + { type: "message_end", message: toolResult }, + { type: "turn_end", message: done, toolResults: [toolResult] }, + ]; +} + +function finalTurn(start: number, text: string, tokens: ReturnType): PiSessionEvent[] { + const streaming = assistant(start, [{ type: "text", text: "" }], "pending"); + const done = assistant(start, [{ type: "text", text }], "stop", { usage: tokens, durationMs: 800, thinkingLevel: "off", rawStopReason: "stop" }); + return [ + { type: "turn_start" }, + { type: "message_start", message: streaming }, + { type: "message_update", message: streaming, assistantMessageEvent: { type: "text_delta", delta: text } }, + { type: "message_end", message: done }, + { type: "turn_end", message: done, toolResults: [] }, + ]; +} + +function promptMessages(): PiSessionEvent[] { + const system = { role: "system", content: SYSTEM_SECRET, sections: {}, toolsAdded: [], timestamp: T0 - 3 }; + const user = { role: "user", content: [{ type: "text", text: PROMPT_SECRET }], timestamp: T0 - 3 }; + return [ + { type: "message_start", message: system }, + { type: "message_end", message: system }, + { type: "message_start", message: user }, + { type: "message_end", message: user }, + ]; +} + +/** agent_start → 2 tool turns (ls, read) → final text turn → agent_end → agent_settled. */ +export function happyPath(): PiSessionEvent[] { + const [turnStart, ...rest] = toolTurn(T0 + 258, "call_ls_1", "ls", { path: ARG_SECRET }, RESULT_SECRET, usage(1182, 75)); + return [ + { type: "agent_start" }, + turnStart, + ...promptMessages(), + ...rest, + ...toolTurn(T0 + 6_490, "call_read_2", "read", { path: "README.md" }, RESULT_SECRET, usage(1000, 48, 244, 0, 0.0125)), + ...finalTurn(T0 + 11_048, OUTPUT_SECRET, usage(1325, 35)), + { type: "agent_end", messages: [], willRetry: false }, + { type: "agent_settled", aborted: false }, + ]; +} + +/** + * A provider error, Pi's automatic retry, then a turn whose tool fails, then + * a final answer. + */ +export function retryAndToolError(): PiSessionEvent[] { + const failedStreaming = assistant(T0 + 100, [], "pending"); + const failed = assistant(T0 + 100, [], "error", { + usage: usage(0, 0), + durationMs: 300, + errorMessage: "429 Too Many Requests: rate limited", + }); + return [ + { type: "agent_start" }, + { type: "turn_start" }, + ...promptMessages(), + { type: "message_start", message: failedStreaming }, + { type: "message_end", message: failed }, + { type: "turn_end", message: failed, toolResults: [] }, + { type: "agent_end", messages: [], willRetry: true }, + { type: "auto_retry_start", attempt: 1, maxAttempts: 3, delayMs: 2000, errorMessage: "429 Too Many Requests: rate limited" }, + { type: "agent_start" }, + ...toolTurn(T0 + 3_000, "call_read_9", "read", { path: ARG_SECRET }, TOOL_ERROR_SECRET, usage(900, 30), true), + ...finalTurn(T0 + 5_000, OUTPUT_SECRET, usage(950, 20)), + { type: "agent_end", messages: [], willRetry: false }, + { type: "auto_retry_end", success: true, attempt: 1 }, + { type: "agent_settled", aborted: false }, + ]; +} + +/** The user aborts while the model is streaming. */ +export function aborted(): PiSessionEvent[] { + const streaming = assistant(T0 + 100, [{ type: "text", text: "partial" }], "pending"); + const stopped = assistant(T0 + 100, [{ type: "text", text: "partial" }], "aborted", { + usage: usage(500, 3), + durationMs: 150, + errorMessage: "Request was aborted", + }); + return [ + { type: "agent_start" }, + { type: "turn_start" }, + ...promptMessages(), + { type: "message_start", message: streaming }, + { type: "message_end", message: stopped }, + { type: "turn_end", message: stopped, toolResults: [] }, + { type: "agent_end", messages: [], willRetry: false }, + { type: "agent_settled", aborted: true }, + ]; +} + +/** A tool (e.g. codemode) that calls another tool through `ctx.executeTool()`. */ +export function nestedTool(): PiSessionEvent[] { + return [ + { type: "agent_start" }, + { type: "turn_start" }, + { type: "tool_execution_start", toolCallId: "outer", toolName: "codemode", args: { code: ARG_SECRET } }, + { type: "tool_execution_start", toolCallId: "inner", toolName: "read", args: { path: ARG_SECRET }, parentToolCallId: "outer" }, + { type: "tool_execution_end", toolCallId: "inner", toolName: "read", result: { content: [{ type: "text", text: RESULT_SECRET }] }, isError: false, durationMs: 5, parentToolCallId: "outer" }, + { type: "tool_execution_end", toolCallId: "outer", toolName: "codemode", result: { content: [{ type: "text", text: RESULT_SECRET }] }, isError: false, durationMs: 9 }, + { type: "turn_end", message: assistant(T0, [], "stop", { usage: usage(1, 1) }), toolResults: [] }, + { type: "agent_end", messages: [], willRetry: false }, + { type: "agent_settled", aborted: false }, + ]; +} + +/** Stops mid-tool: the session is disposed before Pi finishes. */ +export function interrupted(): PiSessionEvent[] { + return happyPath().slice(0, 10); +} + +/** Minimal stand-in for `AgentSession`: `sessionId`, `subscribe()`, `dispose()`. */ +export class FakeSession implements PiSessionLike { + readonly sessionId: string; + sessionName?: string; + private listeners: PiSessionEventListener[] = []; + disposed = false; + + constructor(sessionId: string = SESSION_ID) { + this.sessionId = sessionId; + } + + subscribe(listener: PiSessionEventListener): () => void { + this.listeners.push(listener); + return () => { + this.listeners = this.listeners.filter((l) => l !== listener); + }; + } + + emit(events: PiSessionEvent[]): void { + for (const event of events) { + for (const listener of [...this.listeners]) listener(event); + } + } + + get listenerCount(): number { + return this.listeners.length; + } + + dispose(): void { + this.disposed = true; + this.listeners = []; + } +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/helpers.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/helpers.ts new file mode 100644 index 000000000..a30fb26ab --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/helpers.ts @@ -0,0 +1,78 @@ +import { Context, ContextManager, ROOT_CONTEXT } from "@opentelemetry/api"; +import { + BasicTracerProvider, + InMemorySpanExporter, + ReadableSpan, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; + +export function memoryProvider(): { provider: BasicTracerProvider; exporter: InMemorySpanExporter } { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ spanProcessors: [new SimpleSpanProcessor(exporter)] }); + return { provider, exporter }; +} + +export function byName(spans: ReadableSpan[], name: string): ReadableSpan[] { + return spans.filter((span) => span.name === name); +} + +export function one(spans: ReadableSpan[], name: string): ReadableSpan { + const matches = byName(spans, name); + if (matches.length !== 1) { + throw new Error(`expected one span named ${name}, got ${matches.length}: ${spans.map((s) => s.name).join(", ")}`); + } + return matches[0]; +} + +export function parentId(span: ReadableSpan): string | undefined { + // sdk-trace-base 2.x exposes parentSpanContext; 1.x exposed parentSpanId. + const anySpan = span as unknown as { parentSpanContext?: { spanId: string }; parentSpanId?: string }; + return anySpan.parentSpanContext?.spanId ?? anySpan.parentSpanId; +} + +/** Every attribute value, event attribute and status message, as one string. */ +export function exportedText(spans: ReadableSpan[]): string { + return JSON.stringify( + spans.map((span) => [span.name, span.attributes, span.events.map((e) => e.attributes), span.status]), + ); +} + +export function hrToMs(time: [number, number]): number { + return time[0] * 1000 + Math.round(time[1] / 1e6); +} + +/** Minimal synchronous context manager, so `context.with()` works in tests without extra deps. */ +export class StackContextManager implements ContextManager { + private stack: Context[] = [ROOT_CONTEXT]; + + active(): Context { + return this.stack[this.stack.length - 1]; + } + + with ReturnType>( + ctx: Context, + fn: F, + thisArg?: ThisParameterType, + ...args: A + ): ReturnType { + this.stack.push(ctx); + try { + return fn.call(thisArg, ...args); + } finally { + this.stack.pop(); + } + } + + bind(_ctx: Context, target: T): T { + return target; + } + + enable(): this { + return this; + } + + disable(): this { + this.stack = [ROOT_CONTEXT]; + return this; + } +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/package.test.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/package.test.ts new file mode 100644 index 000000000..03391dd9e --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/package.test.ts @@ -0,0 +1,61 @@ +import { readFileSync } from "fs"; +import { join } from "path"; +import * as api from "../index"; + +const PKG_DIR = join(__dirname, "../.."); + +function manifest(dir: string): Record { + return JSON.parse(readFileSync(join(dir, "package.json"), "utf8")); +} + +describe("package.json", () => { + it("points repository, bugs and homepage at future-agi/traceAI, as traceai_anthropic does", () => { + const ours = manifest(PKG_DIR); + const sibling = manifest(join(PKG_DIR, "../traceai_anthropic")); + expect(ours.repository).toEqual({ + ...(sibling.repository as object), + directory: "typescript/packages/traceai_pi_coding_agent", + }); + expect(ours.bugs).toEqual(sibling.bugs); + expect(ours.homepage).toBe((sibling.homepage as string).replace("traceai_anthropic", "traceai_pi_coding_agent")); + }); + + it("has no runtime dependency on the Pi SDK", () => { + const ours = manifest(PKG_DIR) as { dependencies: Record }; + expect(Object.keys(ours.dependencies).filter((name) => name.includes("pi-coding-agent"))).toEqual([]); + }); + + it("keeps the version in src/version.ts in step with package.json", () => { + expect(api.VERSION).toBe(manifest(PKG_DIR).version); + }); +}); + +describe("public API", () => { + it("exports the entry points the README documents", () => { + for (const name of [ + "PiCodingAgentInstrumentation", + "instrumentPiCodingAgent", + "instrumentSession", + "wrapCreateAgentSession", + "shutdown", + "resolveContentPolicy", + ]) { + expect(typeof (api as Record)[name]).toBe("function"); + } + }); + + it("hides content unless asked", () => { + const saved = [process.env.FI_HIDE_INPUTS, process.env.FI_HIDE_OUTPUTS]; + delete process.env.FI_HIDE_INPUTS; + delete process.env.FI_HIDE_OUTPUTS; + try { + expect(api.resolveContentPolicy()).toMatchObject({ hideInputs: true, hideOutputs: true }); + expect(api.resolveContentPolicy({ hideInputs: false })).toMatchObject({ hideInputs: false, hideOutputs: true }); + } finally { + if (saved[0] === undefined) delete process.env.FI_HIDE_INPUTS; + else process.env.FI_HIDE_INPUTS = saved[0]; + if (saved[1] === undefined) delete process.env.FI_HIDE_OUTPUTS; + else process.env.FI_HIDE_OUTPUTS = saved[1]; + } + }); +}); diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts new file mode 100644 index 000000000..772d4020d --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts @@ -0,0 +1,369 @@ +import { SpanStatusCode, context, trace } from "@opentelemetry/api"; +import { ReadableSpan } from "@opentelemetry/sdk-trace-base"; +import { Attr, PiCodingAgentInstrumentation, SpanNames, wrapCreateAgentSession } from "../index"; +import { promptTokens } from "../sessionTracer"; +import { + ARG_SECRET, + FakeSession, + OUTPUT_SECRET, + PROMPT_SECRET, + RESULT_SECRET, + SESSION_ID, + SYSTEM_SECRET, + TOOL_ERROR_SECRET, + aborted, + happyPath, + interrupted, + nestedTool, + retryAndToolError, +} from "./fixtures/events"; +import { StackContextManager, byName, exportedText, hrToMs, memoryProvider, one, parentId } from "./helpers"; + +const SECRETS = [PROMPT_SECRET, SYSTEM_SECRET, ARG_SECRET, RESULT_SECRET, OUTPUT_SECRET, TOOL_ERROR_SECRET]; + +function run( + events: ReturnType, + config: ConstructorParameters[0] = {}, +): { spans: ReadableSpan[]; session: FakeSession } { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider, ...config }).instrumentSession(session); + session.emit(events); + return { spans: exporter.getFinishedSpans(), session }; +} + +const CONTENT_ON = { traceConfig: { hideInputs: false, hideOutputs: false } }; + +describe("span tree", () => { + it("builds one agent run with turns, LLM calls and tools, all in one trace", () => { + const { spans } = run(happyPath()); + expect(spans.map((s) => s.name).sort()).toEqual( + [ + SpanNames.AGENT_RUN, + "chat gpt-6-luna", + "chat gpt-6-luna", + "chat gpt-6-luna", + "execute_tool ls", + "execute_tool read", + "pi.turn", + "pi.turn", + "pi.turn", + ].sort(), + ); + expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); + + const agent = one(spans, SpanNames.AGENT_RUN); + expect(parentId(agent)).toBeUndefined(); + const turns = byName(spans, "pi.turn"); + for (const turn of turns) expect(parentId(turn)).toBe(agent.spanContext().spanId); + expect(turns.map((t) => t.attributes[Attr.PI_TURN_INDEX]).sort()).toEqual([0, 1, 2]); + + const ls = one(spans, "execute_tool ls"); + const firstTurn = turns.find((t) => t.attributes[Attr.PI_TURN_INDEX] === 0)!; + expect(parentId(ls)).toBe(firstTurn.spanContext().spanId); + for (const llm of byName(spans, "chat gpt-6-luna")) { + expect(turns.map((t) => t.spanContext().spanId)).toContain(parentId(llm)); + } + }); + + it("sets span kinds the Future AGI collector reads, and session.id on every span", () => { + const { spans } = run(happyPath()); + const kind = (name: string) => byName(spans, name).map((s) => [s.attributes[Attr.FI_SPAN_KIND], s.attributes[Attr.GEN_AI_SPAN_KIND]]); + expect(kind(SpanNames.AGENT_RUN)).toEqual([["AGENT", "AGENT"]]); + expect(new Set(kind("pi.turn").flat())).toEqual(new Set(["CHAIN"])); + expect(new Set(kind("chat gpt-6-luna").flat())).toEqual(new Set(["LLM"])); + expect(kind("execute_tool ls")).toEqual([["TOOL", "TOOL"]]); + for (const span of spans) { + expect(span.attributes[Attr.SESSION_ID]).toBe(SESSION_ID); + } + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.attributes[Attr.GEN_AI_OPERATION_NAME]).toBe("invoke_agent"); + expect(agent.attributes[Attr.GEN_AI_AGENT_NAME]).toBe("pi"); + expect(agent.attributes[Attr.GEN_AI_CONVERSATION_ID]).toBe(SESSION_ID); + // No model key on non-LLM spans: the collector would otherwise type them LLM. + for (const span of spans.filter((s) => !s.name.startsWith("chat"))) { + expect(span.attributes[Attr.REQUEST_MODEL]).toBeUndefined(); + } + }); + + it("records model, provider, tokens, cost and timing on LLM spans", () => { + const { spans } = run(happyPath()); + const llms = byName(spans, "chat gpt-6-luna").sort((a, b) => hrToMs(a.startTime) - hrToMs(b.startTime)); + const [first, second, last] = llms; + expect(first.attributes).toMatchObject({ + [Attr.GEN_AI_OPERATION_NAME]: "chat", + [Attr.REQUEST_MODEL]: "gpt-6-luna", + [Attr.RESPONSE_MODEL]: "gpt-6-luna", + [Attr.PROVIDER_NAME]: "cliproxy", + [Attr.PI_API]: "openai-completions", + [Attr.INPUT_TOKENS]: 1182, + [Attr.OUTPUT_TOKENS]: 75, + [Attr.TOTAL_TOKENS]: 1257, + [Attr.PI_STOP_REASON]: "toolUse", + [Attr.PI_TOOL_CALL_COUNT]: 1, + }); + expect(first.attributes[Attr.PI_TOOL_CALL_NAMES]).toEqual(["ls"]); + expect(first.attributes[Attr.FINISH_REASONS]).toEqual(["toolUse"]); + expect(first.attributes[Attr.COST_TOTAL]).toBeUndefined(); + // Pi's usage.input excludes cache reads; the prompt-token count adds them back. + expect(second.attributes[Attr.INPUT_TOKENS]).toBe(1244); + expect(second.attributes[Attr.CACHE_READ_TOKENS]).toBe(244); + expect(second.attributes[Attr.PI_UNCACHED_INPUT_TOKENS]).toBe(1000); + expect(second.attributes[Attr.COST_TOTAL]).toBe(0.0125); + expect(last.attributes[Attr.PI_STOP_REASON]).toBe("stop"); + // The LLM span spans the provider request: message.timestamp .. + durationMs. + expect(hrToMs(first.startTime)).toBe(1_760_000_000_258); + expect(hrToMs(first.endTime)).toBe(1_760_000_000_258 + 900); + for (const llm of llms) expect(llm.status.code).toBe(SpanStatusCode.OK); + + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.attributes).toMatchObject({ + [Attr.PI_RUN_LLM_CALL_COUNT]: 3, + [Attr.PI_RUN_TOOL_CALL_COUNT]: 2, + [Attr.PI_RUN_TURN_COUNT]: 3, + [Attr.PI_RUN_ATTEMPTS]: 1, + [Attr.PI_RUN_INPUT_TOKENS]: 1182 + 1244 + 1325, + [Attr.PI_RUN_OUTPUT_TOKENS]: 75 + 48 + 35, + [Attr.PI_RUN_COST_TOTAL]: 0.0125, + }); + // Run totals are not the promoted gen_ai.usage keys (no double counting). + expect(agent.attributes[Attr.INPUT_TOKENS]).toBeUndefined(); + expect(agent.status.code).toBe(SpanStatusCode.OK); + }); + + it("records tool identity and timing", () => { + const { spans } = run(happyPath()); + const ls = one(spans, "execute_tool ls"); + expect(ls.attributes).toMatchObject({ + [Attr.GEN_AI_OPERATION_NAME]: "execute_tool", + [Attr.GEN_AI_TOOL_NAME]: "ls", + [Attr.TOOL_NAME]: "ls", + [Attr.GEN_AI_TOOL_CALL_ID]: "call_ls_1", + [Attr.TOOL_CALL_ID]: "call_ls_1", + [Attr.PI_TOOL_DURATION_MS]: 20, + [Attr.PI_TOOL_IS_ERROR]: false, + }); + expect(ls.status.code).toBe(SpanStatusCode.OK); + }); + + it("nests a tool called from another tool under the calling tool", () => { + const { spans } = run(nestedTool()); + const outer = one(spans, "execute_tool codemode"); + const inner = one(spans, "execute_tool read"); + expect(parentId(inner)).toBe(outer.spanContext().spanId); + expect(inner.attributes[Attr.PI_TOOL_PARENT_CALL_ID]).toBe("outer"); + }); + + it("parents the run under the caller's active span when a context manager is registered", () => { + const manager = new StackContextManager(); + context.setGlobalContextManager(manager); + try { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + const appSpan = provider.getTracer("app").startSpan("app.request"); + context.with(trace.setSpan(context.active(), appSpan), () => session.emit(happyPath())); + appSpan.end(); + const spans = exporter.getFinishedSpans(); + expect(parentId(one(spans, SpanNames.AGENT_RUN))).toBe(appSpan.spanContext().spanId); + expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); + } finally { + context.disable(); + } + }); + + it("starts a new trace for each run, linked by session.id", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + session.emit(happyPath()); + session.emit(happyPath()); + const agents = byName(exporter.getFinishedSpans(), SpanNames.AGENT_RUN); + expect(agents).toHaveLength(2); + expect(agents[0].spanContext().traceId).not.toBe(agents[1].spanContext().traceId); + expect(agents.map((a) => a.attributes[Attr.SESSION_ID])).toEqual([SESSION_ID, SESSION_ID]); + }); +}); + +describe("errors", () => { + it("marks a failed provider call ERROR, keeps one run across Pi's retry, and marks a failed tool ERROR", () => { + const { spans } = run(retryAndToolError()); + const agents = byName(spans, SpanNames.AGENT_RUN); + expect(agents).toHaveLength(1); + const agent = agents[0]; + expect(agent.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(2); + expect(agent.events.map((e) => e.name)).toEqual(["pi.auto_retry.start", "pi.auto_retry.end"]); + expect(agent.events[0].attributes).toMatchObject({ "pi.attempt": 1, "pi.max_attempts": 3 }); + // The run recovered, so the run itself is OK. + expect(agent.status.code).toBe(SpanStatusCode.OK); + + const failedLlm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + expect(failedLlm.status).toEqual({ code: SpanStatusCode.ERROR, message: "429 Too Many Requests: rate limited" }); + const failedTurn = byName(spans, "pi.turn").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + expect(failedTurn.status.code).toBe(SpanStatusCode.ERROR); + + const tool = one(spans, "execute_tool read"); + expect(tool.status.code).toBe(SpanStatusCode.ERROR); + expect(tool.attributes[Attr.PI_TOOL_IS_ERROR]).toBe(true); + // Tool error text is tool output: hidden by default. + expect(tool.status.message).toBe("tool reported an error (result hidden)"); + }); + + it("puts the tool error text in the status only when outputs are on", () => { + const { spans } = run(retryAndToolError(), CONTENT_ON); + expect(one(spans, "execute_tool read").status.message).toBe(TOOL_ERROR_SECRET); + }); + + it("marks an aborted run and its model call ERROR", () => { + const { spans } = run(aborted()); + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); + expect(agent.attributes[Attr.PI_ABORTED]).toBe(true); + const llm = one(spans, "chat gpt-6-luna"); + expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "Request was aborted" }); + expect(llm.attributes[Attr.PI_ABORTED]).toBe(true); + }); +}); + +describe("content policy", () => { + const ENV_KEYS = ["FI_HIDE_INPUTS", "FI_HIDE_OUTPUTS"]; + const saved: Record = {}; + beforeEach(() => { + for (const key of ENV_KEYS) { + saved[key] = process.env[key]; + delete process.env[key]; + } + }); + afterEach(() => { + for (const key of ENV_KEYS) { + if (saved[key] === undefined) delete process.env[key]; + else process.env[key] = saved[key]; + } + }); + + it("exports no prompt, system prompt, tool argument, tool result or answer text by default", () => { + const text = exportedText(run(retryAndToolError()).spans) + exportedText(run(happyPath()).spans); + for (const secret of SECRETS) expect(text).not.toContain(secret); + expect(text).not.toContain(Attr.INPUT_VALUE); + expect(text).not.toContain(Attr.OUTPUT_VALUE); + }); + + it("exports prompt, tool arguments, tool results and answers when both are opted in", () => { + const { spans } = run(happyPath(), CONTENT_ON); + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.attributes[Attr.INPUT_VALUE]).toBe(PROMPT_SECRET); + expect(agent.attributes[Attr.OUTPUT_VALUE]).toBe(OUTPUT_SECRET); + const ls = one(spans, "execute_tool ls"); + expect(JSON.parse(ls.attributes[Attr.INPUT_VALUE] as string)).toEqual({ path: ARG_SECRET }); + expect(ls.attributes[Attr.OUTPUT_VALUE]).toBe(RESULT_SECRET); + const firstLlm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_TOOL_CALL_COUNT] === 1)!; + expect(firstLlm.attributes[Attr.OUTPUT_VALUE]).toContain("call_ls_1"); + // The system prompt is never exported, even with content on. + expect(exportedText(spans)).not.toContain(SYSTEM_SECRET); + }); + + it("honours inputs and outputs separately", () => { + const inputsOnly = exportedText(run(happyPath(), { traceConfig: { hideInputs: false } }).spans); + expect(inputsOnly).toContain(PROMPT_SECRET); + expect(inputsOnly).toContain(ARG_SECRET); + expect(inputsOnly).not.toContain(RESULT_SECRET); + expect(inputsOnly).not.toContain(OUTPUT_SECRET); + + const outputsOnly = exportedText(run(happyPath(), { traceConfig: { hideOutputs: false } }).spans); + expect(outputsOnly).not.toContain(PROMPT_SECRET); + expect(outputsOnly).not.toContain(ARG_SECRET); + expect(outputsOnly).toContain(RESULT_SECRET); + expect(outputsOnly).toContain(OUTPUT_SECRET); + }); + + it("opts in through FI_HIDE_INPUTS=false / FI_HIDE_OUTPUTS=false only for an explicit false", () => { + process.env.FI_HIDE_INPUTS = "1"; + process.env.FI_HIDE_OUTPUTS = "yes"; + expect(exportedText(run(happyPath()).spans)).not.toContain(PROMPT_SECRET); + process.env.FI_HIDE_INPUTS = " FALSE "; + process.env.FI_HIDE_OUTPUTS = "false"; + const text = exportedText(run(happyPath()).spans); + expect(text).toContain(PROMPT_SECRET); + expect(text).toContain(RESULT_SECRET); + }); + + it("an explicit option beats the environment", () => { + process.env.FI_HIDE_INPUTS = "false"; + const text = exportedText(run(happyPath(), { traceConfig: { hideInputs: true } }).spans); + expect(text).not.toContain(PROMPT_SECRET); + }); + + it("truncates long content to maxContentLength", () => { + const events = happyPath(); + const { spans } = run(events, { ...CONTENT_ON, maxContentLength: 10 }); + const value = one(spans, SpanNames.AGENT_RUN).attributes[Attr.INPUT_VALUE] as string; + expect(value.startsWith(PROMPT_SECRET.slice(0, 10))).toBe(true); + expect(value).toContain("[truncated"); + }); +}); + +describe("lifecycle", () => { + it("ends open spans as incomplete when the session is disposed mid-run", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + session.emit(interrupted()); + expect(exporter.getFinishedSpans().map((s) => s.name)).not.toContain(SpanNames.AGENT_RUN); + session.dispose(); + expect(session.disposed).toBe(true); + const spans = exporter.getFinishedSpans(); + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.attributes[Attr.PI_INCOMPLETE]).toBe(true); + expect(one(spans, "execute_tool ls").attributes[Attr.PI_INCOMPLETE]).toBe(true); + expect(one(spans, "pi.turn").attributes[Attr.PI_INCOMPLETE]).toBe(true); + }); + + it("the stop function unsubscribes and is idempotent per session", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider: provider }); + const stop = instrumentation.instrumentSession(session); + expect(instrumentation.instrumentSession(session)).toBe(stop); + expect(session.listenerCount).toBe(1); + stop(); + stop(); + expect(session.listenerCount).toBe(0); + session.emit(happyPath()); + expect(exporter.getFinishedSpans()).toHaveLength(0); + }); + + it("never throws into Pi on malformed events", () => { + const { provider } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + const junk = [ + { type: "message_start" }, + { type: "message_end", message: null }, + { type: "tool_execution_end", toolCallId: 7 }, + { type: "turn_end", message: "nope", toolResults: "x" }, + { type: "agent_settled" }, + { nottype: true }, + ] as unknown as ReturnType; + expect(() => session.emit(junk)).not.toThrow(); + }); + + it("wrapCreateAgentSession traces the session it returns and passes arguments through", async () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession("wrapped-session"); + const create = jest.fn(async (options: { cwd: string }) => ({ session, extensionsResult: {}, options })); + const traced = wrapCreateAgentSession(create, { tracerProvider: provider }); + const result = await traced({ cwd: "/tmp/x" }); + expect(create).toHaveBeenCalledWith({ cwd: "/tmp/x" }); + expect(result.session).toBe(session); + session.emit(happyPath()); + expect(one(exporter.getFinishedSpans(), SpanNames.AGENT_RUN).attributes[Attr.SESSION_ID]).toBe("wrapped-session"); + }); +}); + +describe("promptTokens", () => { + it("adds cache reads and writes to Pi's uncached input", () => { + expect(promptTokens({ input: 10, cacheRead: 5, cacheWrite: 2 })).toBe(17); + expect(promptTokens({ input: 10 })).toBe(10); + expect(promptTokens({})).toBeUndefined(); + }); +}); diff --git a/typescript/packages/traceai_pi_coding_agent/src/attributes.ts b/typescript/packages/traceai_pi_coding_agent/src/attributes.ts new file mode 100644 index 000000000..2106c86a1 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/attributes.ts @@ -0,0 +1,103 @@ +/** + * Span names and attribute keys for Pi Coding Agent SDK spans. + * + * Standard keys come from `@traceai/fi-semantic-conventions` and the OTel + * GenAI conventions. `pi.*` keys carry Pi-specific facts the standard keys do + * not cover. + */ +import { FISpanKind, SemanticConventions } from "@traceai/fi-semantic-conventions"; + +export const PI_AGENT_NAME = "pi"; + +export const SpanNames = { + /** One agent run: from `agent_start` until `agent_settled` (OTel GenAI `invoke_agent {agent}`). */ + AGENT_RUN: `invoke_agent ${PI_AGENT_NAME}`, + /** One `turn_start` .. `turn_end`: one model response plus the tools it called. */ + TURN: "pi.turn", + /** One assistant message (OTel GenAI `chat {model}`). */ + llm: (model: string | undefined) => (model ? `chat ${model}` : "chat"), + /** One tool execution (OTel GenAI `execute_tool {tool}`). */ + tool: (toolName: string) => `execute_tool ${toolName}`, +} as const; + +export const Attr = { + FI_SPAN_KIND: SemanticConventions.FI_SPAN_KIND, + /** Span kind key the Future AGI collector reads when `gen_ai.*` keys are present. */ + GEN_AI_SPAN_KIND: "gen_ai.span.kind", + GEN_AI_OPERATION_NAME: SemanticConventions.GEN_AI_OPERATION_NAME, + GEN_AI_AGENT_NAME: "gen_ai.agent.name", + GEN_AI_CONVERSATION_ID: SemanticConventions.GEN_AI_CONVERSATION_ID, + SESSION_ID: SemanticConventions.SESSION_ID, + INPUT_VALUE: SemanticConventions.INPUT_VALUE, + INPUT_MIME_TYPE: SemanticConventions.INPUT_MIME_TYPE, + OUTPUT_VALUE: SemanticConventions.OUTPUT_VALUE, + OUTPUT_MIME_TYPE: SemanticConventions.OUTPUT_MIME_TYPE, + + // LLM + REQUEST_MODEL: SemanticConventions.LLM_MODEL_NAME, + RESPONSE_MODEL: SemanticConventions.GEN_AI_RESPONSE_MODEL, + RESPONSE_ID: SemanticConventions.GEN_AI_RESPONSE_ID, + FINISH_REASONS: SemanticConventions.GEN_AI_RESPONSE_FINISH_REASONS, + PROVIDER_NAME: SemanticConventions.LLM_PROVIDER, + INPUT_TOKENS: SemanticConventions.LLM_TOKEN_COUNT_PROMPT, + OUTPUT_TOKENS: SemanticConventions.LLM_TOKEN_COUNT_COMPLETION, + TOTAL_TOKENS: SemanticConventions.LLM_TOKEN_COUNT_TOTAL, + CACHE_READ_TOKENS: SemanticConventions.LLM_TOKEN_COUNT_PROMPT_DETAILS_CACHE_READ, + CACHE_WRITE_TOKENS: SemanticConventions.LLM_TOKEN_COUNT_PROMPT_DETAILS_CACHE_WRITE, + REASONING_TOKENS: SemanticConventions.LLM_TOKEN_COUNT_COMPLETION_DETAILS_REASONING, + /** Cost key the collector promotes. Written only when Pi priced the call (cost > 0). */ + COST_TOTAL: "gen_ai.cost.total", + + // Tool + GEN_AI_TOOL_NAME: "gen_ai.tool.name", + GEN_AI_TOOL_CALL_ID: "gen_ai.tool.call.id", + TOOL_NAME: SemanticConventions.TOOL_NAME, + TOOL_CALL_ID: SemanticConventions.TOOL_CALL_ID, + + // Pi-specific + PI_SESSION_ID: "pi.session.id", + PI_SESSION_NAME: "pi.session.name", + PI_API: "pi.api", + PI_THINKING_LEVEL: "pi.thinking_level", + PI_STOP_REASON: "pi.stop_reason", + PI_ABORTED: "pi.aborted", + /** Pi's own `usage.input`: input tokens that were neither cache reads nor cache writes. */ + PI_UNCACHED_INPUT_TOKENS: "pi.usage.uncached_input_tokens", + PI_TOOL_CALL_NAMES: "pi.llm.tool_call_names", + PI_TOOL_CALL_COUNT: "pi.llm.tool_call_count", + PI_TURN_INDEX: "pi.turn.index", + PI_TURN_TOOL_RESULT_COUNT: "pi.turn.tool_result_count", + PI_TOOL_DURATION_MS: "pi.tool.duration_ms", + PI_TOOL_PARENT_CALL_ID: "pi.tool.parent_call_id", + PI_TOOL_IS_ERROR: "pi.tool.is_error", + PI_RUN_ATTEMPTS: "pi.agent.attempts", + PI_RUN_TURN_COUNT: "pi.agent.turn_count", + PI_RUN_LLM_CALL_COUNT: "pi.agent.llm_call_count", + PI_RUN_TOOL_CALL_COUNT: "pi.agent.tool_call_count", + /** + * Token totals for the whole run. Not the promoted `gen_ai.usage.*` keys, so + * the collector does not count the run's tokens twice (once per LLM span and + * once on the run). + */ + PI_RUN_INPUT_TOKENS: "pi.agent.usage.input_tokens", + PI_RUN_OUTPUT_TOKENS: "pi.agent.usage.output_tokens", + PI_RUN_TOTAL_TOKENS: "pi.agent.usage.total_tokens", + PI_RUN_COST_TOTAL: "pi.agent.cost.total", + /** Set when the span was closed by `dispose()` / `unsubscribe()` before Pi finished it. */ + PI_INCOMPLETE: "pi.incomplete", +} as const; + +export const SpanKind = { + AGENT: FISpanKind.AGENT, + CHAIN: FISpanKind.CHAIN, + LLM: FISpanKind.LLM, + TOOL: FISpanKind.TOOL, +} as const; + +/** Span event names recorded on the run span. */ +export const EventNames = { + AUTO_RETRY_START: "pi.auto_retry.start", + AUTO_RETRY_END: "pi.auto_retry.end", + COMPACTION_START: "pi.compaction.start", + COMPACTION_END: "pi.compaction.end", +} as const; diff --git a/typescript/packages/traceai_pi_coding_agent/src/content.ts b/typescript/packages/traceai_pi_coding_agent/src/content.ts new file mode 100644 index 000000000..772bd55cf --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/content.ts @@ -0,0 +1,72 @@ +/** + * Helpers that turn Pi message content into span text. Only called when the + * content policy allows the value to be exported. + */ +import { PiContentPart, PiMessage } from "./types"; + +export interface ContentPolicy { + /** Drop prompts and tool arguments. */ + hideInputs: boolean; + /** Drop assistant text and tool results. Tool-call arguments follow `hideInputs`. */ + hideOutputs: boolean; + /** Longest exported text value, in characters. Longer values are cut and marked. */ + maxContentLength: number; +} + +export const DEFAULT_MAX_CONTENT_LENGTH = 16_384; + +export function truncate(value: string, maxLength: number): string { + if (maxLength <= 0 || value.length <= maxLength) { + return value; + } + return `${value.slice(0, maxLength)}... [truncated ${value.length - maxLength} chars]`; +} + +/** Text of a message: string content, or its text parts joined. Images become `[image]`. */ +export function messageText(content: PiMessage["content"] | unknown): string { + if (typeof content === "string") { + return content; + } + if (!Array.isArray(content)) { + return ""; + } + const parts: string[] = []; + for (const part of content as PiContentPart[]) { + if (!part || typeof part !== "object") continue; + if (part.type === "text" && typeof part.text === "string") { + parts.push(part.text); + } else if (part.type === "image") { + parts.push("[image]"); + } + } + return parts.join("\n"); +} + +/** Tool calls in an assistant message (`ToolCall` parts). */ +export function toolCalls(message: PiMessage): PiContentPart[] { + if (!Array.isArray(message.content)) { + return []; + } + return message.content.filter((part) => part && typeof part === "object" && part.type === "toolCall"); +} + +/** JSON for span attributes; never throws. */ +export function safeJson(value: unknown): string { + if (typeof value === "string") { + return value; + } + try { + return JSON.stringify(value) ?? ""; + } catch { + return String(value); + } +} + +/** Text of a tool result (`AgentToolResult` has `content` parts, like a message). */ +export function toolResultText(result: unknown): string { + if (result && typeof result === "object" && "content" in result) { + const text = messageText((result as { content?: unknown }).content); + if (text) return text; + } + return typeof result === "string" ? result : safeJson(result); +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/index.ts b/typescript/packages/traceai_pi_coding_agent/src/index.ts new file mode 100644 index 000000000..8fd918c40 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/index.ts @@ -0,0 +1,21 @@ +export { + INSTRUMENTATION_NAME, + PiCodingAgentInstrumentation, + instrumentPiCodingAgent, + instrumentSession, + resolveContentPolicy, + shutdown, + wrapCreateAgentSession, +} from "./instrumentation"; +export type { PiCodingAgentInstrumentationConfig } from "./instrumentation"; +export type { ContentPolicy } from "./content"; +export { Attr, EventNames, PI_AGENT_NAME, SpanKind, SpanNames } from "./attributes"; +export type { + CreateAgentSessionLike, + PiMessage, + PiSessionEvent, + PiSessionEventListener, + PiSessionLike, + PiUsage, +} from "./types"; +export { VERSION } from "./version"; diff --git a/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts b/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts new file mode 100644 index 000000000..a4e5b482c --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts @@ -0,0 +1,244 @@ +/** + * Public entry points: an instrumentation object that traces Pi `AgentSession` + * objects, a wrapper for `createAgentSession()`, and `shutdown()` to flush + * spans before a short script exits. + * + * The Pi SDK is ESM-only and its module namespace cannot be patched, so the + * application either wraps `createAgentSession` or passes each session to + * `instrumentSession()`. + */ +import { TracerProvider, diag, trace } from "@opentelemetry/api"; +import { FITracer, TraceConfigOptions } from "@traceai/fi-core"; +import { ContentPolicy, DEFAULT_MAX_CONTENT_LENGTH } from "./content"; +import { PiSessionTracer } from "./sessionTracer"; +import { CreateAgentSessionLike, PiSessionLike } from "./types"; +import { VERSION } from "./version"; + +export const INSTRUMENTATION_NAME = "@traceai/pi-coding-agent"; + +/** + * fi-core env keys. Read fail-closed: only an explicit `false` (trimmed, any + * case) opts in to content capture. + */ +const FI_HIDE_INPUTS = "FI_HIDE_INPUTS"; +const FI_HIDE_OUTPUTS = "FI_HIDE_OUTPUTS"; + +export interface PiCodingAgentInstrumentationConfig { + /** Provider to create spans with. Defaults to the global provider at call time. */ + tracerProvider?: TracerProvider; + /** + * fi-core trace config. Unlike fi-core, `hideInputs` and `hideOutputs` + * default to `true` here: prompts, tool arguments and tool results of a + * coding agent contain source code, file contents and command output. Opt in + * with `false`, or with `FI_HIDE_INPUTS=false` / `FI_HIDE_OUTPUTS=false`. + */ + traceConfig?: TraceConfigOptions; + /** Longest exported text value, in characters, when content is on. Default 16384. */ + maxContentLength?: number; +} + +/** `false` only for an explicit `false`; any other value (or none) keeps content hidden. */ +function envHide(key: string): boolean { + const value = typeof process !== "undefined" ? process.env?.[key] : undefined; + return !(typeof value === "string" && value.trim().toLowerCase() === "false"); +} + +/** + * Resolve content capture. Precedence: explicit option > fi-core env var > + * this package's default (hidden). + */ +export function resolveContentPolicy( + traceConfig?: TraceConfigOptions, + maxContentLength: number = DEFAULT_MAX_CONTENT_LENGTH, +): ContentPolicy { + return { + hideInputs: traceConfig?.hideInputs ?? envHide(FI_HIDE_INPUTS), + hideOutputs: traceConfig?.hideOutputs ?? envHide(FI_HIDE_OUTPUTS), + maxContentLength, + }; +} + +/** + * Providers passed to an instrumentation, for `shutdown()` with no argument. + * Held weakly so an app that drops a provider lets it be collected. + */ +const knownProviders = new Set>(); + +function liveProviders(): TracerProvider[] { + const live: TracerProvider[] = []; + for (const ref of knownProviders) { + const provider = ref.deref(); + if (provider) live.push(provider); + else knownProviders.delete(ref); + } + return live; +} + +function trackProvider(provider: TracerProvider): void { + if (!liveProviders().includes(provider)) { + knownProviders.add(new WeakRef(provider)); + } +} + +/** Sessions already traced, so a second `instrumentSession()` call is a no-op. */ +const tracedSessions = new WeakMap void>(); + +/** + * Traces Pi Coding Agent SDK sessions. + * + * ```ts + * import { createAgentSession } from "@earendil-works/pi-coding-agent"; + * const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider }); + * const { session } = await instrumentation.wrapCreateAgentSession(createAgentSession)({ ... }); + * // or: instrumentation.instrumentSession(session); + * ``` + */ +export class PiCodingAgentInstrumentation { + private tracerProvider?: TracerProvider; + private readonly traceConfig?: TraceConfigOptions; + private readonly maxContentLength: number; + + constructor(config: PiCodingAgentInstrumentationConfig = {}) { + this.traceConfig = config.traceConfig; + this.maxContentLength = config.maxContentLength ?? DEFAULT_MAX_CONTENT_LENGTH; + if (config.tracerProvider) { + this.setTracerProvider(config.tracerProvider); + } + } + + setTracerProvider(tracerProvider: TracerProvider): void { + this.tracerProvider = tracerProvider; + trackProvider(tracerProvider); + } + + /** The content policy this instrumentation applies. */ + get contentPolicy(): ContentPolicy { + return resolveContentPolicy(this.traceConfig, this.maxContentLength); + } + + /** + * Trace one `AgentSession`. Subscribe before calling `prompt()`. Returns a + * function that stops tracing and ends any open spans. `session.dispose()` + * also ends them. Calling it twice for one session returns the first stop + * function. After an `AgentSessionRuntime` replaces its session (new, resume, + * fork), call this again with the new session. + */ + instrumentSession(session: PiSessionLike): () => void { + const existing = tracedSessions.get(session); + if (existing) return existing; + const policy = this.contentPolicy; + const provider = this.tracerProvider ?? trace.getTracerProvider(); + const tracer = new FITracer({ + tracer: provider.getTracer(INSTRUMENTATION_NAME, VERSION), + traceConfig: { ...this.traceConfig, hideInputs: policy.hideInputs, hideOutputs: policy.hideOutputs }, + }); + const sessionTracer = new PiSessionTracer(session, tracer, policy); + const stop = sessionTracer.attach(); + tracedSessions.set(session, stop); + wrapDispose(session, stop); + return stop; + } + + /** Return a `createAgentSession()` that traces every session it creates. */ + wrapCreateAgentSession(createAgentSession: F): F { + const instrumentation = this; + const wrapped = async function tracedCreateAgentSession(this: unknown, ...args: unknown[]) { + const result = await createAgentSession.apply(this, args); + try { + if (result?.session) instrumentation.instrumentSession(result.session); + } catch (error) { + diag.warn(`${INSTRUMENTATION_NAME}: could not instrument session: ${error}`); + } + return result; + }; + return wrapped as unknown as F; + } + + /** Flush the provider's span processors. Does not shut the provider down. */ + async forceFlush(): Promise { + await flushProvider(this.tracerProvider ?? trace.getTracerProvider()); + } + + /** Same as `forceFlush()`; named for short scripts. */ + async shutdown(): Promise { + await this.forceFlush(); + } +} + +/** End open spans when the app disposes the session; Pi drops listeners on dispose. */ +function wrapDispose(session: PiSessionLike, stop: () => void): void { + const original = session.dispose; + if (typeof original !== "function") return; + try { + Object.defineProperty(session, "dispose", { + configurable: true, + writable: true, + value: function tracedDispose(this: unknown, ...args: unknown[]) { + stop(); + return original.apply(this ?? session, args as []); + }, + }); + } catch (error) { + diag.debug(`${INSTRUMENTATION_NAME}: could not wrap dispose(): ${error}`); + } +} + +/** Python-style convenience constructor. */ +export function instrumentPiCodingAgent( + config: PiCodingAgentInstrumentationConfig = {}, +): PiCodingAgentInstrumentation { + return new PiCodingAgentInstrumentation(config); +} + +/** Trace one session in one call. */ +export function instrumentSession( + session: PiSessionLike, + config: PiCodingAgentInstrumentationConfig = {}, +): () => void { + return new PiCodingAgentInstrumentation(config).instrumentSession(session); +} + +/** Wrap `createAgentSession()` in one call. */ +export function wrapCreateAgentSession( + createAgentSession: F, + config: PiCodingAgentInstrumentationConfig = {}, +): F { + return new PiCodingAgentInstrumentation(config).wrapCreateAgentSession(createAgentSession); +} + +/** + * Flush spans before a short script exits. Flushes `tracerProvider` when + * given; otherwise every provider passed to an instrumentation plus the global + * provider. Export errors are logged, never thrown. + */ +export async function shutdown(tracerProvider?: TracerProvider): Promise { + const providers = tracerProvider ? [tracerProvider] : [...liveProviders(), trace.getTracerProvider()]; + const seen = new Set(); + for (const provider of providers) { + const target = resolveDelegate(provider); + if (seen.has(target)) continue; + seen.add(target); + await flushProvider(target); + } +} + +function resolveDelegate(provider: TracerProvider): TracerProvider { + const maybeProxy = provider as TracerProvider & { + getDelegate?: () => TracerProvider; + forceFlush?: () => Promise; + }; + if (typeof maybeProxy.forceFlush !== "function" && typeof maybeProxy.getDelegate === "function") { + return maybeProxy.getDelegate(); + } + return provider; +} + +async function flushProvider(provider: TracerProvider): Promise { + const target = resolveDelegate(provider) as TracerProvider & { forceFlush?: () => Promise }; + if (typeof target.forceFlush !== "function") return; + try { + await target.forceFlush(); + } catch (error) { + diag.warn(`${INSTRUMENTATION_NAME}: forceFlush failed: ${error}`); + } +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts new file mode 100644 index 000000000..da479345f --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts @@ -0,0 +1,509 @@ +/** + * Turns the event stream of one Pi `AgentSession` into spans. + * + * Tree per agent run (one `session.prompt()`, including Pi's automatic + * retries and compaction): + * + * invoke_agent pi AGENT agent_start .. agent_settled + * ├─ pi.turn CHAIN turn_start .. turn_end + * │ ├─ chat LLM assistant message_start .. message_end + * │ └─ execute_tool TOOL tool_execution_start .. tool_execution_end + * │ └─ execute_tool TOOL (event carries parentToolCallId) + * └─ pi.turn ... + * + * Every span carries `session.id` = `session.sessionId`, so all runs of one + * session group into one Future AGI session. Listeners must never throw into + * Pi: every handler is wrapped. + */ +import { Context, Span, SpanStatusCode, context as otelContext, diag, trace } from "@opentelemetry/api"; +import { FITracer } from "@traceai/fi-core"; +import { MimeType } from "@traceai/fi-semantic-conventions"; +import { Attr, EventNames, PI_AGENT_NAME, SpanKind, SpanNames } from "./attributes"; +import { ContentPolicy, messageText, safeJson, toolCalls, toolResultText, truncate } from "./content"; +import { PiMessage, PiSessionEvent, PiSessionLike, PiUsage } from "./types"; + +const ERROR_TEXT_LIMIT = 1_000; + +interface OpenSpan { + span: Span; + ctx: Context; +} + +interface RunState extends OpenSpan { + attempts: number; + turnCount: number; + llmCount: number; + toolCount: number; + inputTokens: number; + outputTokens: number; + totalTokens: number; + cost: number; + inputText?: string; + lastAssistant?: PiMessage; +} + +interface TurnState extends OpenSpan { + index: number; +} + +function num(value: unknown): number | undefined { + return typeof value === "number" && Number.isFinite(value) ? value : undefined; +} + +function str(value: unknown): string | undefined { + return typeof value === "string" && value.length > 0 ? value : undefined; +} + +/** + * Prompt tokens as OTel counts them: Pi's `usage.input` excludes cache reads + * and cache writes, so add them back. + */ +export function promptTokens(usage: PiUsage): number | undefined { + const input = num(usage.input); + if (input === undefined) return undefined; + return input + (num(usage.cacheRead) ?? 0) + (num(usage.cacheWrite) ?? 0); +} + +export class PiSessionTracer { + private run?: RunState; + private turn?: TurnState; + private llm?: OpenSpan & { message: PiMessage }; + private readonly tools = new Map(); + private pendingInput?: string; + private unsubscribe?: () => void; + private closed = false; + + constructor( + private readonly session: PiSessionLike, + private readonly tracer: FITracer, + private readonly policy: ContentPolicy, + ) {} + + /** Subscribe to the session. Returns the function that stops tracing it. */ + attach(): () => void { + this.unsubscribe = this.session.subscribe((event) => this.onEvent(event)); + return () => this.detach(); + } + + /** Stop listening and end every open span, marked incomplete. */ + detach(): void { + if (this.closed) return; + this.closed = true; + try { + this.unsubscribe?.(); + } catch (error) { + diag.debug(`@traceai/pi-coding-agent: unsubscribe failed: ${error}`); + } + this.endAll(true); + } + + onEvent(event: PiSessionEvent): void { + if (this.closed || !event || typeof event.type !== "string") return; + try { + this.dispatch(event); + } catch (error) { + diag.warn(`@traceai/pi-coding-agent: failed to trace ${event.type}: ${error}`); + } + } + + private dispatch(event: PiSessionEvent): void { + switch (event.type) { + case "agent_start": + this.onAgentStart(); + break; + case "agent_end": + // Pi may retry or continue after agent_end; the run ends at agent_settled. + break; + case "agent_settled": + this.endRun(event.aborted === true); + break; + case "turn_start": + this.onTurnStart(); + break; + case "turn_end": + this.onTurnEnd(event); + break; + case "message_start": + this.onMessageStart(event.message as PiMessage | undefined); + break; + case "message_end": + this.onMessageEnd(event.message as PiMessage | undefined); + break; + case "tool_execution_start": + this.onToolStart(event); + break; + case "tool_execution_end": + this.onToolEnd(event); + break; + case "auto_retry_start": + this.runEvent(EventNames.AUTO_RETRY_START, { + "pi.attempt": num(event.attempt), + "pi.max_attempts": num(event.maxAttempts), + "pi.delay_ms": num(event.delayMs), + }); + break; + case "auto_retry_end": + this.runEvent(EventNames.AUTO_RETRY_END, { + "pi.attempt": num(event.attempt), + "pi.success": event.success === true, + }); + break; + case "compaction_start": + this.runEvent(EventNames.COMPACTION_START, { "pi.reason": str(event.reason) }); + break; + case "compaction_end": + this.runEvent(EventNames.COMPACTION_END, { + "pi.reason": str(event.reason), + "pi.aborted": event.aborted === true, + "pi.will_retry": event.willRetry === true, + }); + break; + default: + // message_update, tool_execution_update, bash_execution_update, queue_update, + // entry_appended, ...: streaming deltas and bookkeeping, not spans. + break; + } + } + + // ---------------------------------------------------------------- spans + + private sessionAttributes(): Record { + const attrs: Record = {}; + const sessionId = str(this.session.sessionId); + if (sessionId) { + attrs[Attr.SESSION_ID] = sessionId; + attrs[Attr.PI_SESSION_ID] = sessionId; + } + return attrs; + } + + private start(name: string, kind: string, parent: Context, attributes: Record, startTime?: number): OpenSpan { + const span = this.tracer.startSpan(name, startTime !== undefined ? { startTime } : {}, parent); + span.setAttributes({ + [Attr.FI_SPAN_KIND]: kind, + [Attr.GEN_AI_SPAN_KIND]: kind, + ...this.sessionAttributes(), + }); + setDefined(span, attributes); + return { span, ctx: trace.setSpan(parent, span) }; + } + + private ensureRun(): RunState { + if (this.run) return this.run; + const sessionId = str(this.session.sessionId); + const open = this.start(SpanNames.AGENT_RUN, SpanKind.AGENT, otelContext.active(), { + [Attr.GEN_AI_OPERATION_NAME]: "invoke_agent", + [Attr.GEN_AI_AGENT_NAME]: PI_AGENT_NAME, + [Attr.GEN_AI_CONVERSATION_ID]: sessionId, + [Attr.PI_SESSION_NAME]: str(this.session.sessionName), + }); + this.run = { + ...open, + attempts: 0, + turnCount: 0, + llmCount: 0, + toolCount: 0, + inputTokens: 0, + outputTokens: 0, + totalTokens: 0, + cost: 0, + inputText: this.pendingInput, + }; + this.pendingInput = undefined; + return this.run; + } + + private onAgentStart(): void { + this.ensureRun().attempts += 1; + } + + private endRun(aborted: boolean, incomplete = false): void { + const run = this.run; + if (!run) return; + this.endTurn(undefined, incomplete); + this.endTools(incomplete); + this.endLlm(undefined, incomplete); + const { span } = run; + if (run.inputText !== undefined && !this.policy.hideInputs) { + span.setAttribute(Attr.INPUT_VALUE, truncate(run.inputText, this.policy.maxContentLength)); + span.setAttribute(Attr.INPUT_MIME_TYPE, MimeType.TEXT); + } + const last = run.lastAssistant; + if (last && !this.policy.hideOutputs) { + span.setAttribute(Attr.OUTPUT_VALUE, truncate(messageText(last.content), this.policy.maxContentLength)); + span.setAttribute(Attr.OUTPUT_MIME_TYPE, MimeType.TEXT); + } + setDefined(span, { + [Attr.PI_RUN_ATTEMPTS]: run.attempts, + [Attr.PI_RUN_TURN_COUNT]: run.turnCount, + [Attr.PI_RUN_LLM_CALL_COUNT]: run.llmCount, + [Attr.PI_RUN_TOOL_CALL_COUNT]: run.toolCount, + [Attr.PI_RUN_INPUT_TOKENS]: run.inputTokens, + [Attr.PI_RUN_OUTPUT_TOKENS]: run.outputTokens, + [Attr.PI_RUN_TOTAL_TOKENS]: run.totalTokens, + [Attr.PI_RUN_COST_TOTAL]: run.cost > 0 ? run.cost : undefined, + [Attr.PI_STOP_REASON]: str(last?.stopReason), + [Attr.PI_ABORTED]: aborted || undefined, + [Attr.PI_INCOMPLETE]: incomplete || undefined, + }); + const failed = last?.stopReason === "error" || last?.stopReason === "aborted"; + if (aborted) { + span.setStatus({ code: SpanStatusCode.ERROR, message: "aborted" }); + } else if (failed) { + span.setStatus({ code: SpanStatusCode.ERROR, message: errorText(last) }); + } else if (!incomplete) { + span.setStatus({ code: SpanStatusCode.OK }); + } + span.end(); + this.run = undefined; + } + + private onTurnStart(): void { + const run = this.ensureRun(); + this.endTurn(undefined, true); + const open = this.start(SpanNames.TURN, SpanKind.CHAIN, run.ctx, { + [Attr.PI_TURN_INDEX]: run.turnCount, + }); + this.turn = { ...open, index: run.turnCount }; + run.turnCount += 1; + } + + private onTurnEnd(event: PiSessionEvent): void { + this.endTurn(event, false); + } + + private endTurn(event: PiSessionEvent | undefined, incomplete: boolean): void { + const turn = this.turn; + if (!turn) return; + const message = event?.message as PiMessage | undefined; + const toolResults = Array.isArray(event?.toolResults) ? (event?.toolResults as unknown[]) : undefined; + setDefined(turn.span, { + [Attr.PI_STOP_REASON]: str(message?.stopReason), + [Attr.PI_TURN_TOOL_RESULT_COUNT]: toolResults?.length, + [Attr.PI_INCOMPLETE]: incomplete || undefined, + }); + if (message?.stopReason === "error" || message?.stopReason === "aborted") { + turn.span.setStatus({ code: SpanStatusCode.ERROR, message: errorText(message) }); + } else if (!incomplete) { + turn.span.setStatus({ code: SpanStatusCode.OK }); + } + turn.span.end(); + this.turn = undefined; + } + + private parentContext(): Context { + return this.turn?.ctx ?? this.ensureRun().ctx; + } + + private onMessageStart(message: PiMessage | undefined): void { + if (!message) return; + if (message.role === "user") { + // The prompt that starts the run, or a steering message inside it. A user + // message seen before agent_start is kept for the run that follows. + const text = messageText(message.content); + if (this.run) { + this.run.inputText = joinInput(this.run.inputText, text); + } else { + this.pendingInput = joinInput(this.pendingInput, text); + } + return; + } + if (message.role !== "assistant") return; + this.endLlm(undefined, true); + const startTime = num(message.timestamp); + const open = this.start(SpanNames.llm(str(message.model)), SpanKind.LLM, this.parentContext(), { + [Attr.GEN_AI_OPERATION_NAME]: "chat", + [Attr.REQUEST_MODEL]: str(message.model), + [Attr.PROVIDER_NAME]: str(message.provider), + [Attr.PI_API]: str(message.api), + }, startTime); + this.llm = { ...open, message }; + } + + private onMessageEnd(message: PiMessage | undefined): void { + if (!message || message.role !== "assistant") return; + if (!this.llm) { + // message_end without message_start (subscribed mid-stream). + this.onMessageStart(message); + } + this.endLlm(message, false); + } + + private endLlm(message: PiMessage | undefined, incomplete: boolean): void { + const llm = this.llm; + if (!llm) return; + this.llm = undefined; + const final = message ?? llm.message; + const { span } = llm; + const usage = (final.usage ?? {}) as PiUsage; + const calls = toolCalls(final); + const input = promptTokens(usage); + setDefined(span, { + [Attr.REQUEST_MODEL]: str(final.model), + [Attr.RESPONSE_MODEL]: str(final.responseModel) ?? str(final.model), + [Attr.RESPONSE_ID]: str(final.responseId), + [Attr.PROVIDER_NAME]: str(final.provider), + [Attr.PI_THINKING_LEVEL]: str(final.thinkingLevel), + [Attr.PI_STOP_REASON]: str(final.stopReason), + [Attr.FINISH_REASONS]: str(final.stopReason) ? [final.stopReason as string] : undefined, + [Attr.PI_TOOL_CALL_COUNT]: calls.length, + [Attr.PI_TOOL_CALL_NAMES]: calls.length ? calls.map((call) => String(call.name ?? "")) : undefined, + [Attr.PI_INCOMPLETE]: incomplete || undefined, + }); + if (!incomplete) { + const output = num(usage.output); + setDefined(span, { + [Attr.INPUT_TOKENS]: input, + [Attr.OUTPUT_TOKENS]: output, + [Attr.TOTAL_TOKENS]: num(usage.totalTokens), + [Attr.CACHE_READ_TOKENS]: num(usage.cacheRead) || undefined, + [Attr.CACHE_WRITE_TOKENS]: num(usage.cacheWrite) || undefined, + [Attr.REASONING_TOKENS]: num(usage.reasoning), + [Attr.PI_UNCACHED_INPUT_TOKENS]: num(usage.input), + [Attr.COST_TOTAL]: (num(usage.cost?.total) ?? 0) > 0 ? usage.cost?.total : undefined, + }); + if (!this.policy.hideOutputs) { + const text = messageText(final.content); + // Tool-call arguments are the tool's input, so they follow hideInputs. + const value = calls.length + ? safeJson({ + text, + toolCalls: calls.map((call) => ({ + id: call.id, + name: call.name, + ...(this.policy.hideInputs ? {} : { arguments: call.arguments }), + })), + }) + : text; + span.setAttribute(Attr.OUTPUT_VALUE, truncate(value, this.policy.maxContentLength)); + span.setAttribute(Attr.OUTPUT_MIME_TYPE, calls.length ? MimeType.JSON : MimeType.TEXT); + } + const run = this.run; + if (run) { + run.llmCount += 1; + run.inputTokens += input ?? 0; + run.outputTokens += output ?? 0; + run.totalTokens += num(usage.totalTokens) ?? 0; + run.cost += num(usage.cost?.total) ?? 0; + run.lastAssistant = final; + } + } + if (final.stopReason === "error" || final.stopReason === "aborted") { + span.setStatus({ code: SpanStatusCode.ERROR, message: errorText(final) }); + if (final.stopReason === "aborted") span.setAttribute(Attr.PI_ABORTED, true); + } else if (!incomplete) { + span.setStatus({ code: SpanStatusCode.OK }); + } + const start = num(final.timestamp); + const duration = num(final.durationMs); + span.end(start !== undefined && duration !== undefined && !incomplete ? start + duration : undefined); + } + + private onToolStart(event: PiSessionEvent): void { + const toolCallId = str(event.toolCallId); + const toolName = str(event.toolName) ?? "unknown"; + const parentCallId = str(event.parentToolCallId); + const parent = (parentCallId && this.tools.get(parentCallId)?.ctx) || this.parentContext(); + const open = this.start(SpanNames.tool(toolName), SpanKind.TOOL, parent, { + [Attr.GEN_AI_OPERATION_NAME]: "execute_tool", + [Attr.GEN_AI_TOOL_NAME]: toolName, + [Attr.TOOL_NAME]: toolName, + [Attr.GEN_AI_TOOL_CALL_ID]: toolCallId, + [Attr.TOOL_CALL_ID]: toolCallId, + [Attr.PI_TOOL_PARENT_CALL_ID]: parentCallId, + }); + if (!this.policy.hideInputs && event.args !== undefined) { + open.span.setAttribute(Attr.INPUT_VALUE, truncate(safeJson(event.args), this.policy.maxContentLength)); + open.span.setAttribute(Attr.INPUT_MIME_TYPE, MimeType.JSON); + } + if (this.run) this.run.toolCount += 1; + // Pi tool call ids are unique within a session; a missing id still gets a span. + this.tools.set(toolCallId ?? `anonymous-${this.tools.size}-${Date.now()}`, open); + } + + private onToolEnd(event: PiSessionEvent): void { + const toolCallId = str(event.toolCallId); + let open = toolCallId ? this.tools.get(toolCallId) : undefined; + if (!open) { + // tool_execution_end without a start (subscribed mid-tool): open one now. + this.onToolStart(event); + open = toolCallId ? this.tools.get(toolCallId) : undefined; + if (!open) return; + } + if (toolCallId) this.tools.delete(toolCallId); + const { span } = open; + const isError = event.isError === true; + setDefined(span, { + [Attr.PI_TOOL_DURATION_MS]: num(event.durationMs), + [Attr.PI_TOOL_IS_ERROR]: isError, + }); + const resultText = this.policy.hideOutputs ? undefined : toolResultText(event.result); + if (resultText !== undefined) { + span.setAttribute(Attr.OUTPUT_VALUE, truncate(resultText, this.policy.maxContentLength)); + span.setAttribute(Attr.OUTPUT_MIME_TYPE, MimeType.TEXT); + } + if (isError) { + span.setStatus({ + code: SpanStatusCode.ERROR, + message: resultText !== undefined ? truncate(resultText, ERROR_TEXT_LIMIT) : "tool reported an error (result hidden)", + }); + } else { + span.setStatus({ code: SpanStatusCode.OK }); + } + span.end(); + } + + private endTools(incomplete: boolean): void { + for (const { span } of this.tools.values()) { + if (incomplete) span.setAttribute(Attr.PI_INCOMPLETE, true); + span.end(); + } + this.tools.clear(); + } + + private runEvent(name: string, attributes: Record): void { + const run = this.run; + if (!run) return; + const defined: Record = {}; + for (const [key, value] of Object.entries(attributes)) { + if (value !== undefined) defined[key] = value as string | number | boolean; + } + run.span.addEvent(name, defined); + } + + private endAll(incomplete: boolean): void { + if (this.run) { + this.endRun(false, incomplete); + } else { + this.endTurn(undefined, incomplete); + this.endTools(incomplete); + this.endLlm(undefined, incomplete); + } + } +} + +function joinInput(first: string | undefined, second: string | undefined): string | undefined { + if (!first) return second; + if (!second) return first; + return `${first}\n\n${second}`; +} + +/** + * Status message for a failed or aborted assistant message. `errorMessage` is + * the provider's error text, not the prompt, so it is kept (truncated). + */ +function errorText(message: PiMessage | undefined): string { + const text = str(message?.errorMessage) ?? (message?.stopReason === "aborted" ? "aborted" : "error"); + return truncate(text, ERROR_TEXT_LIMIT); +} + +function setDefined(span: Span, attributes: Record): void { + for (const [key, value] of Object.entries(attributes)) { + if (value === undefined || value === null) continue; + if (Array.isArray(value)) { + span.setAttribute(key, value.map((item) => String(item))); + } else if (typeof value === "string" || typeof value === "number" || typeof value === "boolean") { + span.setAttribute(key, value); + } + } +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/types.ts b/typescript/packages/traceai_pi_coding_agent/src/types.ts new file mode 100644 index 000000000..aedaab61c --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/types.ts @@ -0,0 +1,74 @@ +/** + * Structural types for the parts of the Pi Coding Agent SDK this package reads. + * + * Shapes follow `@earendil-works/pi-coding-agent` 1.2.0 + * (`dist/core/agent-session.d.ts`), `@earendil-works/pi-agent-core` + * (`AgentEvent`) and `@earendil-works/pi-ai` (`AssistantMessage`, `Usage`). + * They are declared here instead of imported so the package has no runtime or + * type dependency on the SDK: any session object with `sessionId` and + * `subscribe()` works, and fields the SDK adds later are ignored. + */ + +/** One event from `AgentSession.subscribe()`. Only `type` is guaranteed. */ +export interface PiSessionEvent { + type: string; + [key: string]: unknown; +} + +export type PiSessionEventListener = (event: PiSessionEvent) => void; + +/** The `AgentSession` surface this package uses. */ +export interface PiSessionLike { + readonly sessionId: string; + readonly sessionName?: string; + subscribe(listener: PiSessionEventListener): () => void; + dispose?: () => void; +} + +/** The `createAgentSession()` result surface this package uses. */ +export interface CreateAgentSessionResultLike { + session: PiSessionLike; +} + +export type CreateAgentSessionLike = (...args: any[]) => Promise; + +/** `@earendil-works/pi-ai` `Usage`. */ +export interface PiUsage { + input?: number; + output?: number; + cacheRead?: number; + cacheWrite?: number; + reasoning?: number; + totalTokens?: number; + cost?: { total?: number }; +} + +export interface PiContentPart { + type?: string; + text?: string; + name?: string; + id?: string; + arguments?: unknown; +} + +/** Any transcript message (`AgentMessage`). Pi emits system, user, assistant, toolResult and custom roles. */ +export interface PiMessage { + role?: string; + content?: string | PiContentPart[]; + timestamp?: number; + // assistant + api?: string; + provider?: string; + model?: string; + responseModel?: string; + responseId?: string; + thinkingLevel?: string; + usage?: PiUsage; + stopReason?: string; + errorMessage?: string; + durationMs?: number; + // toolResult + toolCallId?: string; + toolName?: string; + isError?: boolean; +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/version.ts b/typescript/packages/traceai_pi_coding_agent/src/version.ts new file mode 100644 index 000000000..52c905c11 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/version.ts @@ -0,0 +1 @@ +export const VERSION = "0.1.0"; diff --git a/typescript/packages/traceai_pi_coding_agent/tsconfig.esm.json b/typescript/packages/traceai_pi_coding_agent/tsconfig.esm.json new file mode 100644 index 000000000..e5d6f9b6c --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/tsconfig.esm.json @@ -0,0 +1,17 @@ +{ + "extends": "../../tsconfig.base.esm.json", + "compilerOptions": { + "outDir": "dist/esm", + "rootDir": "src", + "tsBuildInfoFile": "dist/esm/tsconfig.esm.tsbuildinfo", + "baseUrl": "./src" + }, + "include": [ + "src/**/*.ts" + ], + "references": [], + "exclude": [ + "src/**/*.test.ts", + "src/**/__tests__/**" + ] +} diff --git a/typescript/packages/traceai_pi_coding_agent/tsconfig.esnext.json b/typescript/packages/traceai_pi_coding_agent/tsconfig.esnext.json new file mode 100644 index 000000000..a99f0fe06 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/tsconfig.esnext.json @@ -0,0 +1,17 @@ +{ + "extends": "../../tsconfig.base.esnext.json", + "compilerOptions": { + "outDir": "dist/esnext", + "rootDir": "src", + "tsBuildInfoFile": "dist/esnext/tsconfig.esnext.tsbuildinfo", + "baseUrl": "./src" + }, + "include": [ + "src/**/*.ts" + ], + "references": [], + "exclude": [ + "src/**/*.test.ts", + "src/**/__tests__/**" + ] +} diff --git a/typescript/packages/traceai_pi_coding_agent/tsconfig.json b/typescript/packages/traceai_pi_coding_agent/tsconfig.json new file mode 100644 index 000000000..087f63ee4 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/tsconfig.json @@ -0,0 +1,14 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "outDir": "dist", + "rootDir": ".", + "baseUrl": "./src" + }, + "include": ["src/**/*.ts"], + "exclude": ["src/**/*.test.ts", "src/**/__tests__/**", "test/**/*.ts"], + "references": [ + { "path": "../fi-core" }, + { "path": "../fi-semantic-conventions" } + ] +} diff --git a/typescript/pnpm-lock.yaml b/typescript/pnpm-lock.yaml index c0f93ce14..cdb847538 100644 --- a/typescript/pnpm-lock.yaml +++ b/typescript/pnpm-lock.yaml @@ -1507,6 +1507,43 @@ importers: specifier: ^5.3.3 version: 5.8.3 + packages/traceai_pi_coding_agent: + dependencies: + '@opentelemetry/api': + specifier: ^1.9.0 + version: 1.9.1 + '@traceai/fi-core': + specifier: workspace:^ + version: link:../fi-core + '@traceai/fi-semantic-conventions': + specifier: workspace:^ + version: link:../fi-semantic-conventions + devDependencies: + '@opentelemetry/sdk-trace-base': + specifier: ^2.0.1 + version: 2.7.1(@opentelemetry/api@1.9.1) + '@types/jest': + specifier: ^29.5.14 + version: 29.5.14 + '@types/node': + specifier: ^22.10.10 + version: 22.15.33 + jest: + specifier: ^29.7.0 + version: 29.7.0(@types/node@22.15.33)(ts-node@10.9.2(@types/node@22.15.33)(typescript@5.8.3)) + rimraf: + specifier: ^6.0.1 + version: 6.1.2 + ts-jest: + specifier: ^29.2.5 + version: 29.4.6(@babel/core@7.27.1)(@jest/transform@29.7.0)(@jest/types@29.6.3)(babel-jest@29.7.0(@babel/core@7.27.1))(jest-util@29.7.0)(jest@29.7.0(@types/node@22.15.33)(ts-node@10.9.2(@types/node@22.15.33)(typescript@5.8.3)))(typescript@5.8.3) + tsc-alias: + specifier: ^1.8.10 + version: 1.8.16 + typescript: + specifier: ^5.7.3 + version: 5.8.3 + packages/traceai_pinecone: dependencies: '@opentelemetry/api': From bc6106540f5e07cdfa8880f048f2a4f702873f01 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Thu, 15 Oct 2026 08:21:54 -0700 Subject: [PATCH 2/7] fix(pi-coding-agent): review fixes: old-Pi run end, compaction spans, error privacy (TH-8264) - Run end on Pi before 0.80.5 (no agent_settled): end the run at an agent_end whose willRetry is not true, until the session is seen to settle. Sessions with AgentSession.waitForIdle() (added with agent_settled in 0.80.5) end at agent_settled from the start. Two prompts on old Pi now make two complete traces without dispose(). - Compaction is a pi.compaction CHAIN span with reason, tokens before/after and the summarization call's gen_ai.usage.* and cost; inside a run it is a child (and counts in run totals), session.compact() / pre-prompt compaction is its own one-span trace with the same session.id. - Error privacy: with outputs hidden, error statuses carry only the stop reason, HTTP status and provider error type (error.type set); the raw provider, tool and compaction error text needs hideOutputs: false. Only vendor-style error types are taken from the body, never an echoed "type". - pi.session.name (user-authored) only with hideInputs: false. - finish_reasons maps toolUse to tool_calls and drops pending; children end before parents on dispose; tool end without toolCallId closes its own span. - stop() removes the per-session entry and restores dispose(), so a session can be traced again; Pi's dispose() runs even if ending spans throws. - pi.agent.attempts defined as agent loops; new pi.agent.retry_count. - README: supported Pi versions, always-exported fields, double traces with the Pi extension, register() throw; example/contract use read+ls only. Co-Authored-By: Claude Opus 5.5 --- .../traceai_pi_coding_agent/README.md | 56 ++-- .../contract/real_sdk_session.mjs | 2 +- .../examples/basic.mjs | 2 +- .../src/__tests__/fixtures/events.ts | 114 ++++++- .../src/__tests__/sessionTracer.test.ts | 304 +++++++++++++++++- .../traceai_pi_coding_agent/src/attributes.ts | 28 +- .../src/instrumentation.ts | 60 +++- .../src/sessionTracer.ts | 296 +++++++++++++---- .../traceai_pi_coding_agent/src/types.ts | 18 ++ 9 files changed, 764 insertions(+), 116 deletions(-) diff --git a/typescript/packages/traceai_pi_coding_agent/README.md b/typescript/packages/traceai_pi_coding_agent/README.md index e0a70fbb6..47bcc92a2 100644 --- a/typescript/packages/traceai_pi_coding_agent/README.md +++ b/typescript/packages/traceai_pi_coding_agent/README.md @@ -4,13 +4,14 @@ OpenTelemetry tracing for the [Pi Coding Agent SDK](https://pi.dev/docs/latest/s Alpha. Tested against `@earendil-works/pi-coding-agent` 1.2.0 with recorded event sequences and with a real SDK session (OpenAI-compatible endpoint, built-in `ls` / `read` tools) exported to a local Future AGI collector. -This is the in-process SDK integration. It is not the Pi command-line app (`pi`, traced with a Pi extension) and not the Oh My Pi fork (`omp`). Pi's own model calls are not instrumented a second time: do not also run a provider instrumentor (`@traceai/openai`, `@traceai/anthropic`) on the same process unless you want duplicate model spans. +This is the in-process SDK integration. It is not the Pi command-line app (`pi`, traced with a Pi extension) and not the Oh My Pi fork (`omp`). Pi's own model calls are not instrumented a second time: a provider instrumentor (`@traceai/openai`, `@traceai/anthropic`) in the same process can add duplicate model spans if it instruments the SDK client Pi loads. If the Pi extension for the command-line app is installed in `~/.pi/agent/extensions/` or `.pi/extensions/`, an SDK session that calls `bindExtensions()` loads it too and every prompt is traced twice; see Troubleshooting. | Item | Value | |---|---| | Tested SDK | `@earendil-works/pi-coding-agent` 1.2.0 (Node 22.19+) | -| Package scope | Pi moved from `@mariozechner/*` to `@earendil-works/*` at 0.74.0. Sessions from either scope work: the package reads the session object structurally and has no dependency on the SDK. | -| SDK surface used | `AgentSession.sessionId`, `AgentSession.subscribe()`, `AgentSession.dispose()` | +| Supported Pi versions | **0.80.5 and later (including 1.x):** a run ends at `agent_settled`, after Pi's retries, compaction and continuations. 0.80.5 is the first published version with `agent_settled`. **0.75.4 to 0.80.3:** Pi had no `agent_settled`, so a run ends at the first `agent_end` whose `willRetry` is not `true`. Automatic retries stay in the run; if Pi continues after that `agent_end` (overflow-compaction recovery, messages queued by `agent_end` handlers), the continuation becomes a second trace, and a compaction Pi runs after that `agent_end` is its own one-span trace. **Before 0.75.4 (including every `@mariozechner/pi-coding-agent` release):** `agent_end` has no `willRetry`, so each automatic retry also becomes its own trace. | +| Package scope | Pi moved from `@mariozechner/*` to `@earendil-works/*` at 0.74.0. The package reads the session object structurally and has no dependency on the SDK. | +| SDK surface used | `AgentSession.sessionId`, `AgentSession.sessionName` (only with `hideInputs: false`), `AgentSession.subscribe()`, `AgentSession.dispose()`, and whether `AgentSession.waitForIdle` exists (never called: Pi added it together with `agent_settled`, so its presence means runs end at `agent_settled`) | ## Install @@ -48,42 +49,50 @@ try { `FI_API_KEY` and `FI_SECRET_KEY` are read by fi-core and sent as the `X-Api-Key` / `X-Secret-Key` headers. They are never copied onto spans. -`instrumentSession()` returns a function that stops tracing that session. Calling it twice for one session is a no-op. An `AgentSessionRuntime` replaces its `AgentSession` on new / resume / fork / import; call `instrumentSession(runtime.session)` again after each replacement, as Pi's own docs say for any subscription. +`instrumentSession()` returns a function that stops tracing that session. One instrumentation per session: while a session is traced, a second `instrumentSession()` call (from any instrumentation, whatever its config) returns the first stop function and does not subscribe again. After the stop function, calling `instrumentSession()` again traces the session again. An `AgentSessionRuntime` replaces its `AgentSession` on new / resume / fork / import; call `instrumentSession(runtime.session)` again after each replacement, as Pi's own docs say for any subscription. ## Spans -One trace per agent run (one `session.prompt()`, including Pi's automatic retries and compaction). Every span carries `session.id` = `session.sessionId`, so the runs of one session group together in Future AGI Sessions. +One trace per agent run (one `session.prompt()`, including Pi's automatic retries and the compaction Pi runs after a model call; see "Supported Pi versions" for Pi before 0.80.5). Every span carries `session.id` = `session.sessionId`, so the runs of one session group together in Future AGI Sessions. | Span name | `fi.span.kind` / `gen_ai.span.kind` | From | Parent | |---|---|---|---| -| `invoke_agent pi` | `AGENT` | `agent_start` .. `agent_settled` | the active span at `agent_start` when the app registered a context manager (for example `AsyncLocalStorageContextManager`), else none (a new trace) | +| `invoke_agent pi` | `AGENT` | `agent_start` .. `agent_settled` (Pi before 0.80.5: .. the first `agent_end` without `willRetry: true`) | the active span at `agent_start` when the app registered a context manager (for example `AsyncLocalStorageContextManager`), else none (a new trace) | | `pi.turn` | `CHAIN` | `turn_start` .. `turn_end` | run | | `chat ` | `LLM` | assistant `message_start` .. `message_end` (start/end from the message's `timestamp` + `durationMs`) | turn | | `execute_tool ` | `TOOL` | `tool_execution_start` .. `tool_execution_end` | turn, or the calling tool when the event has `parentToolCallId` | +| `pi.compaction` | `CHAIN` | `compaction_start` .. `compaction_end` | run, when Pi compacts inside a run (threshold or overflow compaction after a model call). A compaction outside a run (`session.compact()`, or the check Pi runs before sending a prompt) is its own one-span trace with the same `session.id` | | Field | Attributes | Source | |---|---|---| -| Session | `session.id`, `pi.session.id`, `gen_ai.conversation.id`, `pi.session.name` | `session.sessionId`, `session.sessionName` | +| Session | `session.id`, `pi.session.id`, `gen_ai.conversation.id`; `pi.session.name` only with `hideInputs: false` (a user-chosen name) | `session.sessionId`, `session.sessionName` | | Model | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.provider.name`, `pi.api`, `pi.thinking_level` | `AssistantMessage` | | Tokens | `gen_ai.usage.input_tokens` (= Pi `input` + `cacheRead` + `cacheWrite`), `output_tokens`, `total_tokens`, `cache_read_tokens`, `cache_write_tokens`, `output_tokens.reasoning`; `pi.usage.uncached_input_tokens` (Pi's own `input`) | `AssistantMessage.usage`, LLM spans only | | Cost | `gen_ai.cost.total` | `usage.cost.total`, only when Pi priced the call (> 0). Models from `models.json` without prices report 0 and get no cost key | -| Finish | `gen_ai.response.finish_reasons`, `pi.stop_reason`, `pi.llm.tool_call_names`, `pi.llm.tool_call_count` | `stopReason`, `toolCall` parts | +| Finish | `gen_ai.response.finish_reasons` (Pi's `stopReason`, with `toolUse` reported as `tool_calls`), `pi.stop_reason` (Pi's own value), `pi.llm.tool_call_names`, `pi.llm.tool_call_count`. Both reasons are left out while a message is still streaming (`pending`) | `stopReason`, `toolCall` parts | | Tools | `gen_ai.tool.name`, `tool.name`, `gen_ai.tool.call.id`, `tool_call.id`, `pi.tool.duration_ms`, `pi.tool.is_error`, `pi.tool.parent_call_id` | tool execution events | -| Run totals | `pi.agent.attempts`, `turn_count`, `llm_call_count`, `tool_call_count`, `pi.agent.usage.*`, `pi.agent.cost.total` | summed LLM spans. Not the promoted `gen_ai.usage.*` keys, so tokens are not counted twice | -| Retries / compaction | span events `pi.auto_retry.start` / `.end`, `pi.compaction.start` / `.end` on the run span | session events | +| Run totals | `pi.agent.attempts` (agent loops in the run: the first, each automatic retry, and each continuation such as overflow-compaction recovery or queued messages), `pi.agent.retry_count` (automatic retries), `pi.agent.turn_count`, `llm_call_count`, `tool_call_count`, `pi.agent.usage.*`, `pi.agent.cost.total` | summed over the run's LLM and compaction spans. Not the promoted `gen_ai.usage.*` keys, so tokens are not counted twice | +| Compaction | `pi.compaction.reason` (`manual`, `threshold`, `overflow`), `pi.compaction.tokens_before`, `pi.compaction.estimated_tokens_after`, `pi.compaction.will_retry`, and the summarization call's `gen_ai.usage.*` / `gen_ai.cost.total` | `compaction_end.result` | +| Retries | span events `pi.auto_retry.start` / `.end` on the run span | session events | +| Errors | `error.type` on ERROR spans: the provider's error type (for example `invalid_request_error`), else the HTTP status, else `error`; `aborted`, `tool_error`, `compaction_error` | see below | Errors: -- An assistant message with `stopReason` `error` or `aborted` makes its LLM span and turn ERROR, with the provider's `errorMessage` as the status message (truncated to 1000 characters). `aborted` also sets `pi.aborted=true`. -- The run span is ERROR when the run ends aborted (`agent_settled.aborted`) or its last model call failed. A failed call that Pi retried successfully leaves the run OK; the failed call's own span stays ERROR. -- A tool result with `isError: true` makes the tool span ERROR. The status message is the tool's error text only when outputs are on; otherwise "tool reported an error (result hidden)". -- `session.dispose()` or the stop function during a run ends every open span with `pi.incomplete=true`. +- An assistant message with `stopReason` `error` or `aborted` makes its LLM span and turn ERROR. By default the status message is built only from the stop reason, an HTTP status at the start of Pi's `errorMessage`, and the provider's error `type`, for example `error: HTTP 400 invalid_request_error` or `aborted`. The provider's raw error text (truncated to 1000 characters) is the status message only with `hideOutputs: false`: gateways and providers echo request fragments, account details and local paths in it. `aborted` also sets `pi.aborted=true`. +- The run span is ERROR when the run ends aborted (`agent_settled.aborted`) or its last model call failed, with the same status message as that call. A failed call that Pi retried successfully leaves the run OK; the failed call's own span stays ERROR. +- A tool result with `isError: true` makes the tool span ERROR. The status message is the tool's error text only with `hideOutputs: false`; otherwise "tool reported an error (result hidden)". +- A compaction that fails is ERROR with "compaction failed" (Pi's error text only with `hideOutputs: false`); an aborted one is ERROR with "aborted". +- `session.dispose()` or the stop function during a run ends every open span with `pi.incomplete=true`, children before their parents. -Not available from the SDK's session events, so not recorded: the provider request payload (the full transcript sent on each model call), streaming deltas and time to first token, per-request HTTP status, and the system prompt. +Not available from the SDK's session events, so not recorded: the provider request payload (the full transcript sent on each model call), streaming deltas and time to first token, a separate per-request HTTP status attribute (only what Pi's error text starts with), the model that wrote a compaction summary, and the system prompt. ## Privacy: content is off by default -Prompts, tool arguments, tool results and assistant text of a coding agent contain source code, file contents and command output. They are **not** recorded unless you opt in: +Prompts, tool arguments, tool results and assistant text of a coding agent contain source code, file contents and command output. They are **not** recorded unless you opt in. + +Always exported: session id, model, provider and API, token counts and cost, stop and finish reasons, tool names and call ids, durations, turn/call counts, compaction reason and token counts, and error statuses built from the stop reason, HTTP status and provider error type. The Pi working directory is never added as an attribute. + +To opt in: ```typescript new PiCodingAgentInstrumentation({ tracerProvider, traceConfig: { hideInputs: false, hideOutputs: false } }); @@ -92,10 +101,12 @@ new PiCodingAgentInstrumentation({ tracerProvider, traceConfig: { hideInputs: fa | Setting | Adds | |---|---| -| `hideInputs: false` | the user prompt (`input.value` on the run span), tool arguments (`input.value` on tool spans), tool-call arguments inside the LLM output | -| `hideOutputs: false` | assistant text (`output.value` on LLM and run spans), tool results (`output.value` on tool spans), tool error text in the status | +| `hideInputs: false` | the user prompt (`input.value` on the run span), tool arguments (`input.value` on tool spans), tool-call arguments inside the LLM output, the session name (`pi.session.name`) | +| `hideOutputs: false` | assistant text (`output.value` on LLM and run spans), tool results (`output.value` on tool spans), the compaction summary (`output.value` on `pi.compaction`), and the raw error text in the status: provider errors (up to 1000 characters), tool errors, compaction errors | + +With content on, tool arguments, tool results and error text usually contain absolute file paths, which include your username. -Precedence: `traceConfig` option, then the env var, then hidden. The env vars fail closed: only `false` (trimmed, any case) turns capture on. Values are cut at `maxContentLength` characters (default 16384). The system prompt is never exported. +Precedence: `traceConfig` option, then the env var, then hidden. The env vars fail closed: only `false` (any case, surrounding spaces ignored) turns capture on. Values are cut at `maxContentLength` characters (default 16384). The system prompt is never exported. ## Flushing @@ -103,10 +114,13 @@ Precedence: `traceConfig` option, then the env var, then hidden. The env vars fa ## Troubleshooting -- **No traces.** `register()` needs `projectName` or `FI_PROJECT_NAME`, and `FI_API_KEY` / `FI_SECRET_KEY`. +- **`FI_PROJECT_NAME is not set`.** `register()` throws without `projectName` or `FI_PROJECT_NAME`. +- **No traces.** Check `FI_API_KEY` / `FI_SECRET_KEY` and, for a self-hosted Future AGI, `FI_BASE_URL`. - **Nothing from a short script.** Await `tracerProvider.shutdown()` before exit. - **Spans stop after `/new`, `/resume` or a fork.** The runtime made a new `AgentSession`; instrument it. -- **Duplicate model spans.** A provider instrumentor is also tracing Pi's HTTP calls. +- **Duplicate model spans.** A provider instrumentor in the same process can add a second model span if it instruments the `openai` / `@anthropic-ai/sdk` client that Pi loads. +- **Every prompt appears as two traces** (one from scope `@traceai/pi-coding-agent`, one from the Pi extension's `futureagi.pi`). The Pi extension for the command-line app is installed in `~/.pi/agent/extensions/` (or `$PI_CODING_AGENT_DIR/extensions/`) or the project's `.pi/extensions/`, and the app calls `session.bindExtensions()`, which starts it. Use one or the other, or create the session with a `DefaultResourceLoader` that leaves the extension out: `noExtensions: true`, or `extensionsOverride: (base) => ({ ...base, extensions: base.extensions.filter((e) => !e.path.endsWith("futureagi-otlp.ts")) })`, then `await loader.reload()` and pass `resourceLoader: loader` to `createAgentSession()`. +- **Runs merge or split on old Pi.** See "Supported Pi versions": before 0.80.5 Pi has no `agent_settled`. ## Tests diff --git a/typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs b/typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs index f0fe41205..9f4ecc623 100644 --- a/typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs +++ b/typescript/packages/traceai_pi_coding_agent/contract/real_sdk_session.mjs @@ -28,7 +28,7 @@ const { session } = await tracedCreateAgentSession({ thinkingLevel: "off", sessionManager: SessionManager.inMemory(cwd), settingsManager: SettingsManager.inMemory({}), - tools: ["read", "bash", "ls"], + tools: ["read", "ls"], }); try { await session.prompt(prompt); diff --git a/typescript/packages/traceai_pi_coding_agent/examples/basic.mjs b/typescript/packages/traceai_pi_coding_agent/examples/basic.mjs index 6da5a8bd9..e2c064683 100644 --- a/typescript/packages/traceai_pi_coding_agent/examples/basic.mjs +++ b/typescript/packages/traceai_pi_coding_agent/examples/basic.mjs @@ -21,7 +21,7 @@ const { session } = await instrumentation.wrapCreateAgentSession(createAgentSess try { await session.prompt("List the files in this folder and read README.md"); - console.log(session.getLastAssistantText()); + console.log(session.getLastAssistantText() ?? ""); } finally { session.dispose(); await tracerProvider.shutdown(); diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts index 1a9677e15..e6783ab75 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts @@ -17,6 +17,13 @@ export const ARG_SECRET = "ARG_SECRET_/tmp/project"; export const RESULT_SECRET = "RESULT_SECRET_README.md_hello.js"; export const OUTPUT_SECRET = "OUTPUT_SECRET_the_title_is_Throwaway"; export const TOOL_ERROR_SECRET = "TOOL_ERROR_SECRET_ENOENT_missing.txt"; +/** A local path with a username, as provider and tool errors carry them. */ +export const LOCAL_USER = "jdoe-secret-user"; +export const LOCAL_PATH = `/Users/${LOCAL_USER}/work/project/PROVIDER_ERROR_SECRET.md`; +export const PROVIDER_ERROR_TEXT = `400 {"type":"error","error":{"type":"invalid_request_error","message":"cannot read ${LOCAL_PATH} for ${LOCAL_USER}"}}`; +export const TOOL_PATH_ERROR = `ENOENT: no such file or directory, access '/Users/${LOCAL_USER}/work/project/TOOL_PATH_SECRET.txt'`; +export const SESSION_NAME_SECRET = "SESSION_NAME_SECRET_fix-billing-bug"; +export const SUMMARY_SECRET = "SUMMARY_SECRET_the_user_asked_about_billing"; const T0 = 1_760_000_000_000; @@ -136,6 +143,10 @@ export function retryAndToolError(): PiSessionEvent[] { durationMs: 300, errorMessage: "429 Too Many Requests: rate limited", }); + // Pi 1.2.0 emits auto_retry_end right after the first successful message_end. + const retried = toolTurn(T0 + 3_000, "call_read_9", "read", { path: ARG_SECRET }, TOOL_ERROR_SECRET, usage(900, 30), true); + const firstEnd = retried.findIndex((event) => event.type === "message_end"); + retried.splice(firstEnd + 1, 0, { type: "auto_retry_end", success: true, attempt: 1 }); return [ { type: "agent_start" }, { type: "turn_start" }, @@ -146,14 +157,101 @@ export function retryAndToolError(): PiSessionEvent[] { { type: "agent_end", messages: [], willRetry: true }, { type: "auto_retry_start", attempt: 1, maxAttempts: 3, delayMs: 2000, errorMessage: "429 Too Many Requests: rate limited" }, { type: "agent_start" }, - ...toolTurn(T0 + 3_000, "call_read_9", "read", { path: ARG_SECRET }, TOOL_ERROR_SECRET, usage(900, 30), true), + ...retried, ...finalTurn(T0 + 5_000, OUTPUT_SECRET, usage(950, 20)), { type: "agent_end", messages: [], willRetry: false }, - { type: "auto_retry_end", success: true, attempt: 1 }, { type: "agent_settled", aborted: false }, ]; } +/** + * A provider error whose text carries a local path and username, and a tool + * error that does too. Pi gives up (no retry). + */ +export function providerAndToolPathErrors(): PiSessionEvent[] { + const toolEvents = toolTurn(T0 + 100, "call_read_p", "read", { path: "missing.txt" }, TOOL_PATH_ERROR, usage(400, 10), true); + const failedStreaming = assistant(T0 + 2_000, [], "pending"); + const failed = assistant(T0 + 2_000, [], "error", { usage: usage(0, 0), durationMs: 120, errorMessage: PROVIDER_ERROR_TEXT }); + return [ + { type: "agent_start" }, + ...toolEvents, + { type: "turn_start" }, + { type: "message_start", message: failedStreaming }, + { type: "message_end", message: failed }, + { type: "turn_end", message: failed, toolResults: [] }, + { type: "agent_end", messages: [], willRetry: false }, + { type: "agent_settled", aborted: false }, + ]; +} + +/** + * One prompt as Pi before 0.80.5 emits it: the same events as `happyPath` + * but no `agent_settled` (it did not exist yet). + */ +export function legacyPrompt(): PiSessionEvent[] { + return happyPath().filter((event) => event.type !== "agent_settled"); +} + +/** A legacy run that Pi retries: agent_end announces the retry with willRetry. */ +export function legacyRetry(): PiSessionEvent[] { + return retryAndToolError().filter((event) => event.type !== "agent_settled"); +} + +/** + * Pi 0.80.5+: after agent_end, a message queued by an agent_end handler makes + * Pi continue the same run (a second agent_start) before agent_settled. + */ +export function continuedRun(): PiSessionEvent[] { + const events = happyPath(); + const settled = events.pop()!; + return [ + ...events, + { type: "agent_start" }, + ...finalTurn(T0 + 20_000, OUTPUT_SECRET, usage(1400, 12)), + { type: "agent_end", messages: [], willRetry: false }, + settled, + ]; +} + +/** Threshold compaction after the run's last model call, inside the run (Pi 1.2.0 order). */ +export function compactionInRun(): PiSessionEvent[] { + const events = happyPath(); + const settled = events.pop()!; + return [ + ...events, + { type: "compaction_start", reason: "threshold" }, + { + type: "compaction_end", + reason: "threshold", + result: { + summary: SUMMARY_SECRET, + firstKeptEntryId: "e1", + tokensBefore: 180_000, + estimatedTokensAfter: 12_000, + usage: usage(5000, 700, 1000, 0, 0.02), + }, + aborted: false, + willRetry: false, + }, + settled, + ]; +} + +/** `session.compact()` between prompts: no run is open. */ +export function manualCompaction(errorMessage?: string): PiSessionEvent[] { + return [ + { type: "compaction_start", reason: "manual" }, + { + type: "compaction_end", + reason: "manual", + result: errorMessage ? undefined : { summary: SUMMARY_SECRET, tokensBefore: 9_000, usage: usage(800, 90) }, + aborted: false, + willRetry: false, + ...(errorMessage ? { errorMessage } : {}), + }, + ]; +} + /** The user aborts while the model is streaming. */ export function aborted(): PiSessionEvent[] { const streaming = assistant(T0 + 100, [{ type: "text", text: "partial" }], "pending"); @@ -194,8 +292,11 @@ export function interrupted(): PiSessionEvent[] { return happyPath().slice(0, 10); } -/** Minimal stand-in for `AgentSession`: `sessionId`, `subscribe()`, `dispose()`. */ -export class FakeSession implements PiSessionLike { +/** + * Pi < 0.80.5 stand-in: `sessionId`, `subscribe()`, `dispose()`, no + * `waitForIdle()` and no `agent_settled`. + */ +export class LegacySession implements PiSessionLike { readonly sessionId: string; sessionName?: string; private listeners: PiSessionEventListener[] = []; @@ -227,3 +328,8 @@ export class FakeSession implements PiSessionLike { this.listeners = []; } } + +/** Pi 1.2.0 stand-in: adds `waitForIdle()`, which came with `agent_settled` in 0.80.5. */ +export class FakeSession extends LegacySession { + async waitForIdle(): Promise {} +} diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts index 772d4020d..393f85a57 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts @@ -1,20 +1,33 @@ import { SpanStatusCode, context, trace } from "@opentelemetry/api"; -import { ReadableSpan } from "@opentelemetry/sdk-trace-base"; +import { BasicTracerProvider, ReadableSpan } from "@opentelemetry/sdk-trace-base"; import { Attr, PiCodingAgentInstrumentation, SpanNames, wrapCreateAgentSession } from "../index"; -import { promptTokens } from "../sessionTracer"; +import { TOOL_ERROR_HIDDEN, errorDetails, finishReason, promptTokens } from "../sessionTracer"; import { ARG_SECRET, FakeSession, + LOCAL_PATH, + LOCAL_USER, + LegacySession, OUTPUT_SECRET, PROMPT_SECRET, + PROVIDER_ERROR_TEXT, RESULT_SECRET, SESSION_ID, + SESSION_NAME_SECRET, + SUMMARY_SECRET, SYSTEM_SECRET, TOOL_ERROR_SECRET, + TOOL_PATH_ERROR, aborted, + compactionInRun, + continuedRun, happyPath, interrupted, + legacyPrompt, + legacyRetry, + manualCompaction, nestedTool, + providerAndToolPathErrors, retryAndToolError, } from "./fixtures/events"; import { StackContextManager, byName, exportedText, hrToMs, memoryProvider, one, parentId } from "./helpers"; @@ -24,9 +37,9 @@ const SECRETS = [PROMPT_SECRET, SYSTEM_SECRET, ARG_SECRET, RESULT_SECRET, OUTPUT function run( events: ReturnType, config: ConstructorParameters[0] = {}, -): { spans: ReadableSpan[]; session: FakeSession } { + session: LegacySession = new FakeSession(), +): { spans: ReadableSpan[]; session: LegacySession } { const { provider, exporter } = memoryProvider(); - const session = new FakeSession(); new PiCodingAgentInstrumentation({ tracerProvider: provider, ...config }).instrumentSession(session); session.emit(events); return { spans: exporter.getFinishedSpans(), session }; @@ -103,7 +116,8 @@ describe("span tree", () => { [Attr.PI_TOOL_CALL_COUNT]: 1, }); expect(first.attributes[Attr.PI_TOOL_CALL_NAMES]).toEqual(["ls"]); - expect(first.attributes[Attr.FINISH_REASONS]).toEqual(["toolUse"]); + // Pi's toolUse is the OTel tool_calls finish reason; pi.stop_reason keeps Pi's value. + expect(first.attributes[Attr.FINISH_REASONS]).toEqual(["tool_calls"]); expect(first.attributes[Attr.COST_TOTAL]).toBeUndefined(); // Pi's usage.input excludes cache reads; the prompt-token count adds them back. expect(second.attributes[Attr.INPUT_TOKENS]).toBe(1244); @@ -111,6 +125,7 @@ describe("span tree", () => { expect(second.attributes[Attr.PI_UNCACHED_INPUT_TOKENS]).toBe(1000); expect(second.attributes[Attr.COST_TOTAL]).toBe(0.0125); expect(last.attributes[Attr.PI_STOP_REASON]).toBe("stop"); + expect(last.attributes[Attr.FINISH_REASONS]).toEqual(["stop"]); // The LLM span spans the provider request: message.timestamp .. + durationMs. expect(hrToMs(first.startTime)).toBe(1_760_000_000_258); expect(hrToMs(first.endTime)).toBe(1_760_000_000_258 + 900); @@ -192,13 +207,16 @@ describe("errors", () => { expect(agents).toHaveLength(1); const agent = agents[0]; expect(agent.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(2); + expect(agent.attributes[Attr.PI_RUN_RETRY_COUNT]).toBe(1); expect(agent.events.map((e) => e.name)).toEqual(["pi.auto_retry.start", "pi.auto_retry.end"]); expect(agent.events[0].attributes).toMatchObject({ "pi.attempt": 1, "pi.max_attempts": 3 }); // The run recovered, so the run itself is OK. expect(agent.status.code).toBe(SpanStatusCode.OK); const failedLlm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; - expect(failedLlm.status).toEqual({ code: SpanStatusCode.ERROR, message: "429 Too Many Requests: rate limited" }); + // The provider's error text is content: by default only the HTTP status leaves the process. + expect(failedLlm.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 429" }); + expect(failedLlm.attributes[Attr.ERROR_TYPE]).toBe("429"); const failedTurn = byName(spans, "pi.turn").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; expect(failedTurn.status.code).toBe(SpanStatusCode.ERROR); @@ -206,12 +224,43 @@ describe("errors", () => { expect(tool.status.code).toBe(SpanStatusCode.ERROR); expect(tool.attributes[Attr.PI_TOOL_IS_ERROR]).toBe(true); // Tool error text is tool output: hidden by default. - expect(tool.status.message).toBe("tool reported an error (result hidden)"); + expect(tool.status.message).toBe(TOOL_ERROR_HIDDEN); + expect(tool.attributes[Attr.ERROR_TYPE]).toBe("tool_error"); }); - it("puts the tool error text in the status only when outputs are on", () => { + it("puts the tool and provider error text in the status only when outputs are on", () => { const { spans } = run(retryAndToolError(), CONTENT_ON); expect(one(spans, "execute_tool read").status.message).toBe(TOOL_ERROR_SECRET); + const failedLlm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + expect(failedLlm.status.message).toBe("429 Too Many Requests: rate limited"); + }); + + it("keeps local paths and usernames in provider and tool errors out of spans by default", () => { + const { spans } = run(providerAndToolPathErrors()); + const text = exportedText(spans); + expect(text).not.toContain(LOCAL_USER); + expect(text).not.toContain("/Users/"); + expect(text).not.toContain("PROVIDER_ERROR_SECRET"); + expect(text).not.toContain("TOOL_PATH_SECRET"); + // Status: stop reason, HTTP status and the provider's error type only. + const llm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 400 invalid_request_error" }); + expect(llm.attributes[Attr.ERROR_TYPE]).toBe("invalid_request_error"); + const turn = byName(spans, "pi.turn").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + expect(turn.status.message).toBe("error: HTTP 400 invalid_request_error"); + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 400 invalid_request_error" }); + expect(one(spans, "execute_tool read").status).toEqual({ code: SpanStatusCode.ERROR, message: TOOL_ERROR_HIDDEN }); + }); + + it("exports the raw provider and tool error text, paths included, only with outputs on", () => { + const text = exportedText(run(providerAndToolPathErrors(), { traceConfig: { hideOutputs: false } }).spans); + expect(text).toContain(JSON.stringify(PROVIDER_ERROR_TEXT).slice(1, -1)); + expect(text).toContain(JSON.stringify(TOOL_PATH_ERROR).slice(1, -1)); + // Inputs alone do not reveal error text. + const inputsOnly = exportedText(run(providerAndToolPathErrors(), { traceConfig: { hideInputs: false } }).spans); + expect(inputsOnly).not.toContain(LOCAL_PATH); + expect(inputsOnly).not.toContain(LOCAL_USER); }); it("marks an aborted run and its model call ERROR", () => { @@ -220,8 +269,152 @@ describe("errors", () => { expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); expect(agent.attributes[Attr.PI_ABORTED]).toBe(true); const llm = one(spans, "chat gpt-6-luna"); - expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "Request was aborted" }); + expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); expect(llm.attributes[Attr.PI_ABORTED]).toBe(true); + expect(llm.attributes[Attr.ERROR_TYPE]).toBe("aborted"); + }); +}); + +describe("errorDetails", () => { + it("builds the default status from the stop reason, HTTP status and provider error type", () => { + expect(errorDetails({ stopReason: "error", errorMessage: PROVIDER_ERROR_TEXT }, false)).toEqual({ + type: "invalid_request_error", + message: "error: HTTP 400 invalid_request_error", + }); + expect(errorDetails({ stopReason: "error", errorMessage: `Connection refused at ${LOCAL_PATH}` }, false)).toEqual({ + type: "error", + message: "error", + }); + expect( + errorDetails({ stopReason: "error", errorMessage: "fetch failed", diagnostics: [{ type: "x", error: { name: "TypeError" } }] }, false), + ).toEqual({ type: "TypeError", message: "error: TypeError" }); + expect(errorDetails({ stopReason: "error", errorMessage: "Request failed with status code 503" }, false)).toEqual({ + type: "503", + message: "error: HTTP 503", + }); + // An echoed request part's "type" is not an error class. + expect( + errorDetails({ stopReason: "error", errorMessage: `502 {"echo":{"type":"text","text":"${LOCAL_PATH}"}}` }, false), + ).toEqual({ type: "502", message: "error: HTTP 502" }); + expect( + errorDetails({ stopReason: "error", errorMessage: `429 {"error":{"message":"slow down","type":"tokens"}}` }, false), + ).toEqual({ type: "tokens", message: "error: HTTP 429 tokens" }); + expect(errorDetails({ stopReason: "aborted", errorMessage: "Request was aborted" }, false)).toEqual({ + type: "aborted", + message: "aborted", + }); + expect(errorDetails(undefined, false)).toEqual({ type: "error", message: "error" }); + }); + + it("uses the raw text, truncated, when outputs are shown", () => { + expect(errorDetails({ stopReason: "error", errorMessage: PROVIDER_ERROR_TEXT }, true).message).toBe(PROVIDER_ERROR_TEXT); + const long = errorDetails({ stopReason: "error", errorMessage: "x".repeat(5000) }, true).message; + expect(long.startsWith("x".repeat(1000))).toBe(true); + expect(long).toContain("[truncated 4000 chars]"); + }); +}); + +describe("Pi versions without agent_settled (before 0.80.5)", () => { + it("ends each run at agent_end: two prompts make two complete traces before dispose", () => { + const { provider, exporter } = memoryProvider(); + const session = new LegacySession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + session.emit(legacyPrompt()); + session.emit(legacyPrompt()); + // No dispose: both runs are already exported, complete and OK. + const spans = exporter.getFinishedSpans(); + const agents = byName(spans, SpanNames.AGENT_RUN); + expect(agents).toHaveLength(2); + expect(agents[0].spanContext().traceId).not.toBe(agents[1].spanContext().traceId); + for (const agent of agents) { + expect(agent.status.code).toBe(SpanStatusCode.OK); + expect(agent.attributes[Attr.PI_INCOMPLETE]).toBeUndefined(); + expect(agent.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(1); + expect(agent.attributes[Attr.PI_RUN_LLM_CALL_COUNT]).toBe(3); + expect(agent.attributes[Attr.SESSION_ID]).toBe(SESSION_ID); + } + // Every child is in its own run's trace. + for (const span of spans) { + expect(agents.map((a) => a.spanContext().traceId)).toContain(span.spanContext().traceId); + } + expect(spans.filter((s) => s.spanContext().traceId === agents[0].spanContext().traceId)).toHaveLength(9); + expect(spans.filter((s) => s.spanContext().traceId === agents[1].spanContext().traceId)).toHaveLength(9); + }); + + it("keeps a run open across an agent_end that announces a retry", () => { + const { spans } = run(legacyRetry(), {}, new LegacySession()); + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(2); + expect(agent.attributes[Attr.PI_RUN_RETRY_COUNT]).toBe(1); + expect(agent.status.code).toBe(SpanStatusCode.OK); + expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); + }); + + it("switches to agent_settled once a session without waitForIdle() is seen to settle", () => { + const { provider, exporter } = memoryProvider(); + const session = new LegacySession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + session.emit(happyPath()); // ends at agent_end; agent_settled then marks the session as settling + session.emit(continuedRun()); // the continuation after agent_end stays in the same run + const agents = byName(exporter.getFinishedSpans(), SpanNames.AGENT_RUN); + expect(agents).toHaveLength(2); + expect(agents[1].attributes[Attr.PI_RUN_ATTEMPTS]).toBe(2); + expect(agents[1].attributes[Attr.PI_RUN_LLM_CALL_COUNT]).toBe(4); + }); + + it("keeps a continuation after agent_end in the run when the session has waitForIdle() (Pi 0.80.5+)", () => { + const { spans } = run(continuedRun()); + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(2); + expect(agent.attributes[Attr.PI_RUN_TURN_COUNT]).toBe(4); + expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); + }); +}); + +describe("compaction", () => { + it("records compaction inside a run as a CHAIN child with its own token usage", () => { + const { spans } = run(compactionInRun()); + const agent = one(spans, SpanNames.AGENT_RUN); + const compaction = one(spans, SpanNames.COMPACTION); + expect(parentId(compaction)).toBe(agent.spanContext().spanId); + expect(compaction.attributes).toMatchObject({ + [Attr.FI_SPAN_KIND]: "CHAIN", + [Attr.GEN_AI_SPAN_KIND]: "CHAIN", + [Attr.SESSION_ID]: SESSION_ID, + [Attr.PI_COMPACTION_REASON]: "threshold", + [Attr.PI_COMPACTION_TOKENS_BEFORE]: 180_000, + [Attr.PI_COMPACTION_TOKENS_AFTER]: 12_000, + [Attr.PI_COMPACTION_WILL_RETRY]: false, + [Attr.INPUT_TOKENS]: 6000, + [Attr.OUTPUT_TOKENS]: 700, + [Attr.CACHE_READ_TOKENS]: 1000, + [Attr.COST_TOTAL]: 0.02, + }); + expect(compaction.attributes[Attr.REQUEST_MODEL]).toBeUndefined(); + expect(compaction.status.code).toBe(SpanStatusCode.OK); + // Run totals include the compaction model call. + expect(agent.attributes[Attr.PI_RUN_INPUT_TOKENS]).toBe(1182 + 1244 + 1325 + 6000); + expect(agent.attributes[Attr.PI_RUN_COST_TOTAL]).toBeCloseTo(0.0325); + // The summary is model-written text about the conversation: hidden by default. + expect(exportedText(spans)).not.toContain(SUMMARY_SECRET); + expect(one(run(compactionInRun(), CONTENT_ON).spans, SpanNames.COMPACTION).attributes[Attr.OUTPUT_VALUE]).toBe(SUMMARY_SECRET); + }); + + it("records a compaction outside a run (session.compact(), before a prompt) as its own trace", () => { + const { spans } = run(manualCompaction()); + expect(spans.map((s) => s.name)).toEqual([SpanNames.COMPACTION]); + const [compaction] = spans; + expect(parentId(compaction)).toBeUndefined(); + expect(compaction.attributes).toMatchObject({ [Attr.PI_COMPACTION_REASON]: "manual", [Attr.SESSION_ID]: SESSION_ID, [Attr.INPUT_TOKENS]: 800 }); + }); + + it("marks a failed compaction ERROR without its error text by default", () => { + const failure = `summarization failed reading ${LOCAL_PATH}`; + const hidden = one(run(manualCompaction(failure)).spans, SpanNames.COMPACTION); + expect(hidden.status).toEqual({ code: SpanStatusCode.ERROR, message: "compaction failed" }); + expect(exportedText([hidden])).not.toContain(LOCAL_USER); + const shown = one(run(manualCompaction(failure), CONTENT_ON).spans, SpanNames.COMPACTION); + expect(shown.status.message).toBe(failure); }); }); @@ -248,6 +441,19 @@ describe("content policy", () => { expect(text).not.toContain(Attr.OUTPUT_VALUE); }); + it("exports the session name (user-authored) only when inputs are shown", () => { + const named = () => { + const session = new FakeSession(); + session.sessionName = SESSION_NAME_SECRET; + return session; + }; + const hidden = run(happyPath(), {}, named()).spans; + expect(exportedText(hidden)).not.toContain(SESSION_NAME_SECRET); + expect(one(hidden, SpanNames.AGENT_RUN).attributes[Attr.PI_SESSION_NAME]).toBeUndefined(); + const shown = run(happyPath(), { traceConfig: { hideInputs: false } }, named()).spans; + expect(one(shown, SpanNames.AGENT_RUN).attributes[Attr.PI_SESSION_NAME]).toBe(SESSION_NAME_SECRET); + }); + it("exports prompt, tool arguments, tool results and answers when both are opted in", () => { const { spans } = run(happyPath(), CONTENT_ON); const agent = one(spans, SpanNames.AGENT_RUN); @@ -318,6 +524,76 @@ describe("lifecycle", () => { expect(one(spans, "pi.turn").attributes[Attr.PI_INCOMPLETE]).toBe(true); }); + it("ends children before their parents and drops a streaming finish reason when disposed mid-call", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + // Stop while the nested tool runs and while a model call streams in the same turn. + const events = nestedTool().slice(0, 4); + events.push({ type: "message_start", message: { role: "assistant", model: "m", stopReason: "pending", content: [] } }); + session.emit(events); + session.dispose(); + const order = exporter.getFinishedSpans().map((s) => s.name); + expect(order).toEqual(["execute_tool read", "execute_tool codemode", "chat m", "pi.turn", SpanNames.AGENT_RUN]); + const llm = one(exporter.getFinishedSpans(), "chat m"); + expect(llm.attributes[Attr.FINISH_REASONS]).toBeUndefined(); + expect(llm.attributes[Attr.PI_STOP_REASON]).toBeUndefined(); + expect(llm.attributes[Attr.PI_INCOMPLETE]).toBe(true); + }); + + it("ends a tool span without a toolCallId at its end event, with the end event's status", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + session.emit([ + { type: "agent_start" }, + { type: "turn_start" }, + { type: "tool_execution_start", toolName: "bash", args: {} }, + { type: "tool_execution_end", toolName: "bash", result: { content: [] }, isError: true, durationMs: 7 }, + ]); + const tools = byName(exporter.getFinishedSpans(), "execute_tool bash"); + expect(tools).toHaveLength(1); + expect(tools[0].status.code).toBe(SpanStatusCode.ERROR); + expect(tools[0].attributes[Attr.PI_TOOL_IS_ERROR]).toBe(true); + expect(tools[0].attributes[Attr.PI_TOOL_DURATION_MS]).toBe(7); + session.dispose(); + expect(byName(exporter.getFinishedSpans(), "execute_tool bash")).toHaveLength(1); + }); + + it("traces the session again after the stop function, and restores dispose()", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + const instrumentation = new PiCodingAgentInstrumentation({ tracerProvider: provider }); + const originalDispose = session.dispose; + const stop = instrumentation.instrumentSession(session); + expect(session.dispose).not.toBe(originalDispose); + stop(); + expect(session.dispose).toBe(originalDispose); + expect(Object.prototype.hasOwnProperty.call(session, "dispose")).toBe(false); + const again = instrumentation.instrumentSession(session); + expect(again).not.toBe(stop); + expect(session.listenerCount).toBe(1); + session.emit(happyPath()); + expect(byName(exporter.getFinishedSpans(), SpanNames.AGENT_RUN)).toHaveLength(1); + }); + + it("still runs Pi's dispose() when ending the spans throws", () => { + const throwing = { + onStart: () => {}, + onEnd: () => { + throw new Error("span processor failed"); + }, + forceFlush: async () => {}, + shutdown: async () => {}, + }; + const badProvider = new BasicTracerProvider({ spanProcessors: [throwing] }); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: badProvider }).instrumentSession(session); + session.emit(interrupted()); + expect(() => session.dispose()).not.toThrow(); + expect(session.disposed).toBe(true); + }); + it("the stop function unsubscribes and is idempotent per session", () => { const { provider, exporter } = memoryProvider(); const session = new FakeSession(); @@ -360,6 +636,16 @@ describe("lifecycle", () => { }); }); +describe("finishReason", () => { + it("maps Pi stop reasons to OTel finish reasons", () => { + expect(finishReason("toolUse")).toBe("tool_calls"); + expect(finishReason("stop")).toBe("stop"); + expect(finishReason("length")).toBe("length"); + expect(finishReason("pending")).toBeUndefined(); + expect(finishReason(undefined)).toBeUndefined(); + }); +}); + describe("promptTokens", () => { it("adds cache reads and writes to Pi's uncached input", () => { expect(promptTokens({ input: 10, cacheRead: 5, cacheWrite: 2 })).toBe(17); diff --git a/typescript/packages/traceai_pi_coding_agent/src/attributes.ts b/typescript/packages/traceai_pi_coding_agent/src/attributes.ts index 2106c86a1..ee9293a51 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/attributes.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/attributes.ts @@ -10,7 +10,11 @@ import { FISpanKind, SemanticConventions } from "@traceai/fi-semantic-convention export const PI_AGENT_NAME = "pi"; export const SpanNames = { - /** One agent run: from `agent_start` until `agent_settled` (OTel GenAI `invoke_agent {agent}`). */ + /** + * One agent run: from `agent_start` until `agent_settled` (OTel GenAI + * `invoke_agent {agent}`). Pi before 0.80.5 has no `agent_settled`; the run + * then ends at an `agent_end` without `willRetry`. + */ AGENT_RUN: `invoke_agent ${PI_AGENT_NAME}`, /** One `turn_start` .. `turn_end`: one model response plus the tools it called. */ TURN: "pi.turn", @@ -18,6 +22,8 @@ export const SpanNames = { llm: (model: string | undefined) => (model ? `chat ${model}` : "chat"), /** One tool execution (OTel GenAI `execute_tool {tool}`). */ tool: (toolName: string) => `execute_tool ${toolName}`, + /** One context compaction: `compaction_start` .. `compaction_end`. */ + COMPACTION: "pi.compaction", } as const; export const Attr = { @@ -32,6 +38,8 @@ export const Attr = { INPUT_MIME_TYPE: SemanticConventions.INPUT_MIME_TYPE, OUTPUT_VALUE: SemanticConventions.OUTPUT_VALUE, OUTPUT_MIME_TYPE: SemanticConventions.OUTPUT_MIME_TYPE, + /** OTel `error.type`: the error class on ERROR spans (provider error type, HTTP status, `aborted`, `tool_error`). */ + ERROR_TYPE: "error.type", // LLM REQUEST_MODEL: SemanticConventions.LLM_MODEL_NAME, @@ -70,19 +78,29 @@ export const Attr = { PI_TOOL_DURATION_MS: "pi.tool.duration_ms", PI_TOOL_PARENT_CALL_ID: "pi.tool.parent_call_id", PI_TOOL_IS_ERROR: "pi.tool.is_error", + /** + * Agent loops (`agent_start` events) in the run: the first, each automatic + * retry, and each continuation (overflow-compaction recovery, queued messages). + */ PI_RUN_ATTEMPTS: "pi.agent.attempts", + /** Automatic retries in the run (`auto_retry_start` events). */ + PI_RUN_RETRY_COUNT: "pi.agent.retry_count", PI_RUN_TURN_COUNT: "pi.agent.turn_count", PI_RUN_LLM_CALL_COUNT: "pi.agent.llm_call_count", PI_RUN_TOOL_CALL_COUNT: "pi.agent.tool_call_count", /** - * Token totals for the whole run. Not the promoted `gen_ai.usage.*` keys, so - * the collector does not count the run's tokens twice (once per LLM span and - * once on the run). + * Token totals for the whole run (its LLM and compaction spans). Not the + * promoted `gen_ai.usage.*` keys, so the collector does not count the run's + * tokens twice (once per child span and once on the run). */ PI_RUN_INPUT_TOKENS: "pi.agent.usage.input_tokens", PI_RUN_OUTPUT_TOKENS: "pi.agent.usage.output_tokens", PI_RUN_TOTAL_TOKENS: "pi.agent.usage.total_tokens", PI_RUN_COST_TOTAL: "pi.agent.cost.total", + PI_COMPACTION_REASON: "pi.compaction.reason", + PI_COMPACTION_TOKENS_BEFORE: "pi.compaction.tokens_before", + PI_COMPACTION_TOKENS_AFTER: "pi.compaction.estimated_tokens_after", + PI_COMPACTION_WILL_RETRY: "pi.compaction.will_retry", /** Set when the span was closed by `dispose()` / `unsubscribe()` before Pi finished it. */ PI_INCOMPLETE: "pi.incomplete", } as const; @@ -98,6 +116,4 @@ export const SpanKind = { export const EventNames = { AUTO_RETRY_START: "pi.auto_retry.start", AUTO_RETRY_END: "pi.auto_retry.end", - COMPACTION_START: "pi.compaction.start", - COMPACTION_END: "pi.compaction.end", } as const; diff --git a/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts b/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts index a4e5b482c..190ecc27d 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts @@ -80,7 +80,7 @@ function trackProvider(provider: TracerProvider): void { } } -/** Sessions already traced, so a second `instrumentSession()` call is a no-op. */ +/** Sessions being traced, so a second `instrumentSession()` call is a no-op until the first is stopped. */ const tracedSessions = new WeakMap void>(); /** @@ -119,9 +119,13 @@ export class PiCodingAgentInstrumentation { /** * Trace one `AgentSession`. Subscribe before calling `prompt()`. Returns a * function that stops tracing and ends any open spans. `session.dispose()` - * also ends them. Calling it twice for one session returns the first stop - * function. After an `AgentSessionRuntime` replaces its session (new, resume, - * fork), call this again with the new session. + * also ends them. + * + * One instrumentation per session: while a session is traced, calling this + * again (from any instrumentation) returns the first stop function and does + * not subscribe twice. After the stop function (or `dispose()`), calling it + * again subscribes again. After an `AgentSessionRuntime` replaces its + * session (new, resume, fork), call this with the new session. */ instrumentSession(session: PiSessionLike): () => void { const existing = tracedSessions.get(session); @@ -133,9 +137,15 @@ export class PiCodingAgentInstrumentation { traceConfig: { ...this.traceConfig, hideInputs: policy.hideInputs, hideOutputs: policy.hideOutputs }, }); const sessionTracer = new PiSessionTracer(session, tracer, policy); - const stop = sessionTracer.attach(); + sessionTracer.attach(); + let unwrapDispose = (): void => {}; + const stop = (): void => { + if (tracedSessions.get(session) === stop) tracedSessions.delete(session); + unwrapDispose(); + sessionTracer.detach(); + }; tracedSessions.set(session, stop); - wrapDispose(session, stop); + unwrapDispose = wrapDispose(session, stop); return stop; } @@ -165,22 +175,38 @@ export class PiCodingAgentInstrumentation { } } -/** End open spans when the app disposes the session; Pi drops listeners on dispose. */ -function wrapDispose(session: PiSessionLike, stop: () => void): void { +/** + * End open spans when the app disposes the session; Pi drops listeners on + * dispose. Returns a function that puts the original `dispose` back. Pi's own + * `dispose()` always runs, even if ending the spans throws. + */ +function wrapDispose(session: PiSessionLike, stop: () => void): () => void { const original = session.dispose; - if (typeof original !== "function") return; + if (typeof original !== "function") return () => {}; + const ownDescriptor = Object.getOwnPropertyDescriptor(session, "dispose"); + const tracedDispose = function tracedDispose(this: unknown, ...args: unknown[]) { + try { + stop(); + } catch (error) { + diag.warn(`${INSTRUMENTATION_NAME}: failed to end spans on dispose(): ${error}`); + } + return original.apply(this ?? session, args as []); + }; try { - Object.defineProperty(session, "dispose", { - configurable: true, - writable: true, - value: function tracedDispose(this: unknown, ...args: unknown[]) { - stop(); - return original.apply(this ?? session, args as []); - }, - }); + Object.defineProperty(session, "dispose", { configurable: true, writable: true, value: tracedDispose }); } catch (error) { diag.debug(`${INSTRUMENTATION_NAME}: could not wrap dispose(): ${error}`); + return () => {}; } + return () => { + if (session.dispose !== tracedDispose) return; + try { + if (ownDescriptor) Object.defineProperty(session, "dispose", ownDescriptor); + else delete (session as { dispose?: unknown }).dispose; + } catch (error) { + diag.debug(`${INSTRUMENTATION_NAME}: could not restore dispose(): ${error}`); + } + }; } /** Python-style convenience constructor. */ diff --git a/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts index da479345f..d90914411 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts @@ -2,15 +2,26 @@ * Turns the event stream of one Pi `AgentSession` into spans. * * Tree per agent run (one `session.prompt()`, including Pi's automatic - * retries and compaction): + * retries, continuations and compaction inside the run): * - * invoke_agent pi AGENT agent_start .. agent_settled + * invoke_agent pi AGENT agent_start .. agent_settled (see "Run end" below) * ├─ pi.turn CHAIN turn_start .. turn_end * │ ├─ chat LLM assistant message_start .. message_end * │ └─ execute_tool TOOL tool_execution_start .. tool_execution_end * │ └─ execute_tool TOOL (event carries parentToolCallId) + * ├─ pi.compaction CHAIN compaction_start .. compaction_end * └─ pi.turn ... * + * A compaction outside a run (before a prompt, or `session.compact()`) is its + * own one-span trace with the same `session.id`. + * + * Run end: Pi 0.80.5 and later emit `agent_settled` once Pi will not continue + * on its own; the run ends there. Older Pi has no `agent_settled`, so until + * the session is known to settle, the run ends at an `agent_end` that does not + * announce a retry (`willRetry !== true`). Pi added `AgentSession.waitForIdle()` + * in the same release as `agent_settled`, so a session that has it is treated + * as settling from the start. + * * Every span carries `session.id` = `session.sessionId`, so all runs of one * session group into one Future AGI session. Listeners must never throw into * Pi: every handler is wrapped. @@ -20,10 +31,13 @@ import { FITracer } from "@traceai/fi-core"; import { MimeType } from "@traceai/fi-semantic-conventions"; import { Attr, EventNames, PI_AGENT_NAME, SpanKind, SpanNames } from "./attributes"; import { ContentPolicy, messageText, safeJson, toolCalls, toolResultText, truncate } from "./content"; -import { PiMessage, PiSessionEvent, PiSessionLike, PiUsage } from "./types"; +import { PiCompactionResult, PiMessage, PiSessionEvent, PiSessionLike, PiUsage } from "./types"; const ERROR_TEXT_LIMIT = 1_000; +/** Status message of a failed tool when outputs are hidden. */ +export const TOOL_ERROR_HIDDEN = "tool reported an error (result hidden)"; + interface OpenSpan { span: Span; ctx: Context; @@ -31,6 +45,7 @@ interface OpenSpan { interface RunState extends OpenSpan { attempts: number; + retries: number; turnCount: number; llmCount: number; toolCount: number; @@ -64,20 +79,97 @@ export function promptTokens(usage: PiUsage): number | undefined { return input + (num(usage.cacheRead) ?? 0) + (num(usage.cacheWrite) ?? 0); } +/** + * `gen_ai.response.finish_reasons` value for a Pi `stopReason`. Pi normalizes + * provider reasons to `stop`, `length`, `toolUse`, `error`, `aborted`; + * `toolUse` becomes the OTel `tool_calls`. `pending` (still streaming) has none. + */ +export function finishReason(stopReason: string | undefined): string | undefined { + if (!stopReason || stopReason === "pending") return undefined; + return stopReason === "toolUse" ? "tool_calls" : stopReason; +} + +const HTTP_STATUS = /^\s*(?:HTTP\s*)?([45]\d\d)\b|\bstatus(?:\s*code)?\s*[:=]?\s*([45]\d\d)\b/i; +const PROVIDER_ERROR_TYPE = /"type"\s*:\s*"([A-Za-z][A-Za-z0-9_.-]{0,63})"/g; +const IDENTIFIER = /^[A-Za-z][A-Za-z0-9_.-]{0,63}$/; + +/** + * Error class and status message for a failed or aborted assistant message. + * + * Pi's `errorMessage` is the provider's raw error text. Gateways and + * providers echo request fragments, account details and local paths in it, + * so it is content: it is the status message only when outputs are shown. + * Otherwise the message is built from the stop reason, the HTTP status at the + * start of the text and the provider's error `type` (an identifier such as + * `invalid_request_error`), for example `error: HTTP 400 invalid_request_error`. + */ +export function errorDetails( + message: PiMessage | undefined, + showText: boolean, +): { type: string; message: string } { + if (message?.stopReason === "aborted") { + const raw = str(message.errorMessage); + return { type: "aborted", message: showText && raw ? truncate(raw, ERROR_TEXT_LIMIT) : "aborted" }; + } + const raw = str(message?.errorMessage); + const match = raw ? HTTP_STATUS.exec(raw) : null; + const status = match ? match[1] ?? match[2] : undefined; + const providerType = raw ? providerErrorType(raw) : undefined; + const errorClass = providerType ?? diagnosticErrorName(message) ?? status ?? "error"; + if (showText && raw) return { type: errorClass, message: truncate(raw, ERROR_TEXT_LIMIT) }; + const detail = [status ? `HTTP ${status}` : undefined, providerType ?? diagnosticErrorName(message)] + .filter((part): part is string => part !== undefined) + .join(" "); + return { type: errorClass, message: detail ? `error: ${detail}` : "error" }; +} + +/** + * Provider error types that do not end in `error` (OpenAI quota and rate-limit + * kinds, Anthropic's 413). Any other `"type"` value must end in `error` / + * `Error`, so an echoed request part such as `"type":"text"` is never taken + * for the error class. + */ +const UNSUFFIXED_ERROR_TYPES = new Set(["insufficient_quota", "request_too_large", "tokens", "requests"]); + +function providerErrorType(raw: string): string | undefined { + for (const match of raw.matchAll(PROVIDER_ERROR_TYPE)) { + const type = match[1]; + if (type === "error") continue; + if (/error$/i.test(type) || UNSUFFIXED_ERROR_TYPES.has(type)) return type; + } + return undefined; +} + +/** The thrown error's class name Pi recorded on the message, when it is more than `Error`. */ +function diagnosticErrorName(message: PiMessage | undefined): string | undefined { + const diagnostics = Array.isArray(message?.diagnostics) ? message!.diagnostics : []; + for (let i = diagnostics.length - 1; i >= 0; i--) { + const name = diagnostics[i]?.error?.name; + if (typeof name === "string" && name !== "Error" && IDENTIFIER.test(name)) return name; + } + return undefined; +} + export class PiSessionTracer { private run?: RunState; private turn?: TurnState; private llm?: OpenSpan & { message: PiMessage }; + private compaction?: OpenSpan; private readonly tools = new Map(); + private anonymousToolSeq = 0; private pendingInput?: string; private unsubscribe?: () => void; private closed = false; + /** True once the session is known to emit `agent_settled` (Pi 0.80.5+). */ + private settles: boolean; constructor( private readonly session: PiSessionLike, private readonly tracer: FITracer, private readonly policy: ContentPolicy, - ) {} + ) { + this.settles = typeof (session as { waitForIdle?: unknown }).waitForIdle === "function"; + } /** Subscribe to the session. Returns the function that stops tracing it. */ attach(): () => void { @@ -85,7 +177,7 @@ export class PiSessionTracer { return () => this.detach(); } - /** Stop listening and end every open span, marked incomplete. */ + /** Stop listening and end every open span, marked incomplete. Never throws. */ detach(): void { if (this.closed) return; this.closed = true; @@ -94,7 +186,11 @@ export class PiSessionTracer { } catch (error) { diag.debug(`@traceai/pi-coding-agent: unsubscribe failed: ${error}`); } - this.endAll(true); + try { + this.endAll(true); + } catch (error) { + diag.warn(`@traceai/pi-coding-agent: failed to end open spans: ${error}`); + } } onEvent(event: PiSessionEvent): void { @@ -112,9 +208,13 @@ export class PiSessionTracer { this.onAgentStart(); break; case "agent_end": - // Pi may retry or continue after agent_end; the run ends at agent_settled. + // Pi 0.80.5+ may retry or continue after agent_end; the run ends at + // agent_settled. Older Pi never settles: end at an agent_end that does + // not announce a retry. + if (!this.settles && event.willRetry !== true) this.endRun(false); break; case "agent_settled": + this.settles = true; this.endRun(event.aborted === true); break; case "turn_start": @@ -136,6 +236,7 @@ export class PiSessionTracer { this.onToolEnd(event); break; case "auto_retry_start": + if (this.run) this.run.retries += 1; this.runEvent(EventNames.AUTO_RETRY_START, { "pi.attempt": num(event.attempt), "pi.max_attempts": num(event.maxAttempts), @@ -149,14 +250,10 @@ export class PiSessionTracer { }); break; case "compaction_start": - this.runEvent(EventNames.COMPACTION_START, { "pi.reason": str(event.reason) }); + this.onCompactionStart(event); break; case "compaction_end": - this.runEvent(EventNames.COMPACTION_END, { - "pi.reason": str(event.reason), - "pi.aborted": event.aborted === true, - "pi.will_retry": event.willRetry === true, - }); + this.onCompactionEnd(event); break; default: // message_update, tool_execution_update, bash_execution_update, queue_update, @@ -188,6 +285,11 @@ export class PiSessionTracer { return { span, ctx: trace.setSpan(parent, span) }; } + private setError(span: Span, details: { type: string; message: string }): void { + span.setAttribute(Attr.ERROR_TYPE, details.type); + span.setStatus({ code: SpanStatusCode.ERROR, message: details.message }); + } + private ensureRun(): RunState { if (this.run) return this.run; const sessionId = str(this.session.sessionId); @@ -195,11 +297,13 @@ export class PiSessionTracer { [Attr.GEN_AI_OPERATION_NAME]: "invoke_agent", [Attr.GEN_AI_AGENT_NAME]: PI_AGENT_NAME, [Attr.GEN_AI_CONVERSATION_ID]: sessionId, - [Attr.PI_SESSION_NAME]: str(this.session.sessionName), + // The session name is user-authored text (/name, --name, setSessionName()). + [Attr.PI_SESSION_NAME]: this.policy.hideInputs ? undefined : str(this.session.sessionName), }); this.run = { ...open, attempts: 0, + retries: 0, turnCount: 0, llmCount: 0, toolCount: 0, @@ -217,12 +321,18 @@ export class PiSessionTracer { this.ensureRun().attempts += 1; } + /** End the run's open children, deepest first. */ + private endChildren(incomplete: boolean): void { + this.endTools(incomplete); + this.endLlm(undefined, incomplete); + this.endTurn(undefined, incomplete); + this.endCompaction(incomplete); + } + private endRun(aborted: boolean, incomplete = false): void { const run = this.run; if (!run) return; - this.endTurn(undefined, incomplete); - this.endTools(incomplete); - this.endLlm(undefined, incomplete); + this.endChildren(incomplete); const { span } = run; if (run.inputText !== undefined && !this.policy.hideInputs) { span.setAttribute(Attr.INPUT_VALUE, truncate(run.inputText, this.policy.maxContentLength)); @@ -235,6 +345,7 @@ export class PiSessionTracer { } setDefined(span, { [Attr.PI_RUN_ATTEMPTS]: run.attempts, + [Attr.PI_RUN_RETRY_COUNT]: run.retries, [Attr.PI_RUN_TURN_COUNT]: run.turnCount, [Attr.PI_RUN_LLM_CALL_COUNT]: run.llmCount, [Attr.PI_RUN_TOOL_CALL_COUNT]: run.toolCount, @@ -242,15 +353,15 @@ export class PiSessionTracer { [Attr.PI_RUN_OUTPUT_TOKENS]: run.outputTokens, [Attr.PI_RUN_TOTAL_TOKENS]: run.totalTokens, [Attr.PI_RUN_COST_TOTAL]: run.cost > 0 ? run.cost : undefined, - [Attr.PI_STOP_REASON]: str(last?.stopReason), + [Attr.PI_STOP_REASON]: last?.stopReason === "pending" ? undefined : str(last?.stopReason), [Attr.PI_ABORTED]: aborted || undefined, [Attr.PI_INCOMPLETE]: incomplete || undefined, }); const failed = last?.stopReason === "error" || last?.stopReason === "aborted"; if (aborted) { - span.setStatus({ code: SpanStatusCode.ERROR, message: "aborted" }); + this.setError(span, { type: "aborted", message: "aborted" }); } else if (failed) { - span.setStatus({ code: SpanStatusCode.ERROR, message: errorText(last) }); + this.setError(span, errorDetails(last, !this.policy.hideOutputs)); } else if (!incomplete) { span.setStatus({ code: SpanStatusCode.OK }); } @@ -278,12 +389,12 @@ export class PiSessionTracer { const message = event?.message as PiMessage | undefined; const toolResults = Array.isArray(event?.toolResults) ? (event?.toolResults as unknown[]) : undefined; setDefined(turn.span, { - [Attr.PI_STOP_REASON]: str(message?.stopReason), + [Attr.PI_STOP_REASON]: message?.stopReason === "pending" ? undefined : str(message?.stopReason), [Attr.PI_TURN_TOOL_RESULT_COUNT]: toolResults?.length, [Attr.PI_INCOMPLETE]: incomplete || undefined, }); if (message?.stopReason === "error" || message?.stopReason === "aborted") { - turn.span.setStatus({ code: SpanStatusCode.ERROR, message: errorText(message) }); + this.setError(turn.span, errorDetails(message, !this.policy.hideOutputs)); } else if (!incomplete) { turn.span.setStatus({ code: SpanStatusCode.OK }); } @@ -338,30 +449,24 @@ export class PiSessionTracer { const usage = (final.usage ?? {}) as PiUsage; const calls = toolCalls(final); const input = promptTokens(usage); + // A message still streaming (`pending`) has no finish reason yet. + const stopReason = incomplete || final.stopReason === "pending" ? undefined : str(final.stopReason); + const finish = finishReason(stopReason); setDefined(span, { [Attr.REQUEST_MODEL]: str(final.model), [Attr.RESPONSE_MODEL]: str(final.responseModel) ?? str(final.model), [Attr.RESPONSE_ID]: str(final.responseId), [Attr.PROVIDER_NAME]: str(final.provider), [Attr.PI_THINKING_LEVEL]: str(final.thinkingLevel), - [Attr.PI_STOP_REASON]: str(final.stopReason), - [Attr.FINISH_REASONS]: str(final.stopReason) ? [final.stopReason as string] : undefined, + [Attr.PI_STOP_REASON]: stopReason, + [Attr.FINISH_REASONS]: finish ? [finish] : undefined, [Attr.PI_TOOL_CALL_COUNT]: calls.length, [Attr.PI_TOOL_CALL_NAMES]: calls.length ? calls.map((call) => String(call.name ?? "")) : undefined, [Attr.PI_INCOMPLETE]: incomplete || undefined, }); if (!incomplete) { const output = num(usage.output); - setDefined(span, { - [Attr.INPUT_TOKENS]: input, - [Attr.OUTPUT_TOKENS]: output, - [Attr.TOTAL_TOKENS]: num(usage.totalTokens), - [Attr.CACHE_READ_TOKENS]: num(usage.cacheRead) || undefined, - [Attr.CACHE_WRITE_TOKENS]: num(usage.cacheWrite) || undefined, - [Attr.REASONING_TOKENS]: num(usage.reasoning), - [Attr.PI_UNCACHED_INPUT_TOKENS]: num(usage.input), - [Attr.COST_TOTAL]: (num(usage.cost?.total) ?? 0) > 0 ? usage.cost?.total : undefined, - }); + setUsage(span, usage); if (!this.policy.hideOutputs) { const text = messageText(final.content); // Tool-call arguments are the tool's input, so they follow hideInputs. @@ -389,7 +494,7 @@ export class PiSessionTracer { } } if (final.stopReason === "error" || final.stopReason === "aborted") { - span.setStatus({ code: SpanStatusCode.ERROR, message: errorText(final) }); + this.setError(span, errorDetails(final, !this.policy.hideOutputs)); if (final.stopReason === "aborted") span.setAttribute(Attr.PI_ABORTED, true); } else if (!incomplete) { span.setStatus({ code: SpanStatusCode.OK }); @@ -399,7 +504,7 @@ export class PiSessionTracer { span.end(start !== undefined && duration !== undefined && !incomplete ? start + duration : undefined); } - private onToolStart(event: PiSessionEvent): void { + private onToolStart(event: PiSessionEvent): string { const toolCallId = str(event.toolCallId); const toolName = str(event.toolName) ?? "unknown"; const parentCallId = str(event.parentToolCallId); @@ -418,19 +523,25 @@ export class PiSessionTracer { } if (this.run) this.run.toolCount += 1; // Pi tool call ids are unique within a session; a missing id still gets a span. - this.tools.set(toolCallId ?? `anonymous-${this.tools.size}-${Date.now()}`, open); + const key = toolCallId ?? `${anonymousPrefix(toolName)}${this.anonymousToolSeq++}`; + this.tools.set(key, open); + return key; } private onToolEnd(event: PiSessionEvent): void { const toolCallId = str(event.toolCallId); - let open = toolCallId ? this.tools.get(toolCallId) : undefined; - if (!open) { + let key: string | undefined = toolCallId; + if (!toolCallId) { + // No id: match the oldest open id-less span of the same tool. + const prefix = anonymousPrefix(str(event.toolName) ?? "unknown"); + key = [...this.tools.keys()].find((candidate) => candidate.startsWith(prefix)); + } + if (!key || !this.tools.has(key)) { // tool_execution_end without a start (subscribed mid-tool): open one now. - this.onToolStart(event); - open = toolCallId ? this.tools.get(toolCallId) : undefined; - if (!open) return; + key = this.onToolStart(event); } - if (toolCallId) this.tools.delete(toolCallId); + const open = this.tools.get(key)!; + this.tools.delete(key); const { span } = open; const isError = event.isError === true; setDefined(span, { @@ -443,9 +554,10 @@ export class PiSessionTracer { span.setAttribute(Attr.OUTPUT_MIME_TYPE, MimeType.TEXT); } if (isError) { - span.setStatus({ - code: SpanStatusCode.ERROR, - message: resultText !== undefined ? truncate(resultText, ERROR_TEXT_LIMIT) : "tool reported an error (result hidden)", + // The error text is the tool's result (paths, command output): content. + this.setError(span, { + type: "tool_error", + message: resultText !== undefined ? truncate(resultText, ERROR_TEXT_LIMIT) : TOOL_ERROR_HIDDEN, }); } else { span.setStatus({ code: SpanStatusCode.OK }); @@ -454,13 +566,76 @@ export class PiSessionTracer { } private endTools(incomplete: boolean): void { - for (const { span } of this.tools.values()) { + // Newest first, so a nested tool ends before the tool that called it. + for (const { span } of [...this.tools.values()].reverse()) { if (incomplete) span.setAttribute(Attr.PI_INCOMPLETE, true); span.end(); } this.tools.clear(); } + private onCompactionStart(event: PiSessionEvent): void { + this.endCompaction(true); + // Inside a run (threshold or overflow compaction after a model call) it is + // a child of the run; before a prompt or from session.compact() it is its + // own trace. + const parent = this.run?.ctx ?? otelContext.active(); + this.compaction = this.start(SpanNames.COMPACTION, SpanKind.CHAIN, parent, { + [Attr.PI_COMPACTION_REASON]: str(event.reason), + }); + } + + private onCompactionEnd(event: PiSessionEvent): void { + if (!this.compaction) this.onCompactionStart(event); + const { span } = this.compaction!; + this.compaction = undefined; + const result = (event.result && typeof event.result === "object" ? event.result : undefined) as + | PiCompactionResult + | undefined; + setDefined(span, { + [Attr.PI_COMPACTION_REASON]: str(event.reason), + [Attr.PI_COMPACTION_TOKENS_BEFORE]: num(result?.tokensBefore), + [Attr.PI_COMPACTION_TOKENS_AFTER]: num(result?.estimatedTokensAfter), + [Attr.PI_COMPACTION_WILL_RETRY]: event.willRetry === true, + [Attr.PI_ABORTED]: event.aborted === true || undefined, + }); + const usage = result?.usage && typeof result.usage === "object" ? (result.usage as PiUsage) : undefined; + if (usage) { + setUsage(span, usage); + const run = this.run; + if (run) { + run.inputTokens += promptTokens(usage) ?? 0; + run.outputTokens += num(usage.output) ?? 0; + run.totalTokens += num(usage.totalTokens) ?? 0; + run.cost += num(usage.cost?.total) ?? 0; + } + } + if (!this.policy.hideOutputs && str(result?.summary)) { + span.setAttribute(Attr.OUTPUT_VALUE, truncate(result!.summary as string, this.policy.maxContentLength)); + span.setAttribute(Attr.OUTPUT_MIME_TYPE, MimeType.TEXT); + } + const errorMessage = str(event.errorMessage); + if (event.aborted === true) { + this.setError(span, { type: "aborted", message: "aborted" }); + } else if (errorMessage) { + this.setError(span, { + type: "compaction_error", + message: this.policy.hideOutputs ? "compaction failed" : truncate(errorMessage, ERROR_TEXT_LIMIT), + }); + } else { + span.setStatus({ code: SpanStatusCode.OK }); + } + span.end(); + } + + private endCompaction(incomplete: boolean): void { + const compaction = this.compaction; + if (!compaction) return; + this.compaction = undefined; + if (incomplete) compaction.span.setAttribute(Attr.PI_INCOMPLETE, true); + compaction.span.end(); + } + private runEvent(name: string, attributes: Record): void { const run = this.run; if (!run) return; @@ -475,26 +650,33 @@ export class PiSessionTracer { if (this.run) { this.endRun(false, incomplete); } else { - this.endTurn(undefined, incomplete); - this.endTools(incomplete); - this.endLlm(undefined, incomplete); + this.endChildren(incomplete); } } } +function anonymousPrefix(toolName: string): string { + return `\u0000anonymous:${toolName}:`; +} + function joinInput(first: string | undefined, second: string | undefined): string | undefined { if (!first) return second; if (!second) return first; return `${first}\n\n${second}`; } -/** - * Status message for a failed or aborted assistant message. `errorMessage` is - * the provider's error text, not the prompt, so it is kept (truncated). - */ -function errorText(message: PiMessage | undefined): string { - const text = str(message?.errorMessage) ?? (message?.stopReason === "aborted" ? "aborted" : "error"); - return truncate(text, ERROR_TEXT_LIMIT); +/** Promoted token and cost keys from a Pi `Usage`. */ +function setUsage(span: Span, usage: PiUsage): void { + setDefined(span, { + [Attr.INPUT_TOKENS]: promptTokens(usage), + [Attr.OUTPUT_TOKENS]: num(usage.output), + [Attr.TOTAL_TOKENS]: num(usage.totalTokens), + [Attr.CACHE_READ_TOKENS]: num(usage.cacheRead) || undefined, + [Attr.CACHE_WRITE_TOKENS]: num(usage.cacheWrite) || undefined, + [Attr.REASONING_TOKENS]: num(usage.reasoning), + [Attr.PI_UNCACHED_INPUT_TOKENS]: num(usage.input), + [Attr.COST_TOTAL]: (num(usage.cost?.total) ?? 0) > 0 ? usage.cost?.total : undefined, + }); } function setDefined(span: Span, attributes: Record): void { diff --git a/typescript/packages/traceai_pi_coding_agent/src/types.ts b/typescript/packages/traceai_pi_coding_agent/src/types.ts index aedaab61c..09c6eaead 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/types.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/types.ts @@ -20,9 +20,16 @@ export type PiSessionEventListener = (event: PiSessionEvent) => void; /** The `AgentSession` surface this package uses. */ export interface PiSessionLike { readonly sessionId: string; + /** User-chosen display name. Exported only when inputs are shown. */ readonly sessionName?: string; subscribe(listener: PiSessionEventListener): () => void; dispose?: () => void; + /** + * Present from Pi 0.80.5, the release that added `agent_settled`. Only its + * presence is checked (never called): it tells the tracer that runs end at + * `agent_settled`. + */ + waitForIdle?: () => Promise; } /** The `createAgentSession()` result surface this package uses. */ @@ -66,9 +73,20 @@ export interface PiMessage { usage?: PiUsage; stopReason?: string; errorMessage?: string; + rawStopReason?: string; + /** Errors Pi recorded while producing the message; only `error.name` is read. */ + diagnostics?: Array<{ type?: string; error?: { name?: string } }>; durationMs?: number; // toolResult toolCallId?: string; toolName?: string; isError?: boolean; } + +/** `compaction_end.result` (`CompactionResult`). `summary` is model-written text: content. */ +export interface PiCompactionResult { + summary?: string; + tokensBefore?: number; + estimatedTokensAfter?: number; + usage?: PiUsage; +} From 8067247c0c3eb8291919ef7b2312e398dcedff5a Mon Sep 17 00:00:00 2001 From: Nikhil Pareek Date: Thu, 15 Oct 2026 13:48:20 -0700 Subject: [PATCH 3/7] fix(pi-coding-agent): end the run when a legacy Pi retry is cancelled On Pi 0.75.4 to 0.80.3 (no agent_settled), session.abort() or abortRetry() during the retry backoff emits auto_retry_end with success:false after an agent_end that announced the retry, and no further agent_start. The run stayed open, so the next prompt joined the failed run's trace and the run ended OK. End the run at that auto_retry_end when the session does not settle. Sessions that settle (Pi 0.80.5+) still end at agent_settled. Tests: a legacy cancelled retry gives a complete ERROR run before the next prompt and a separate trace for that prompt; exhausted retries still end at the last agent_end; a settling session waits for agent_settled. Checked against real Pi 0.75.4 and 0.80.3 via both abort() and abortRetry(). Docs: README error-status wording now names the error class-name fallback and the "status code NNN" form; the contract README uses the self-hosted collector port 4318; test fixtures use a neutral model and provider name. Co-Authored-By: Claude Opus 5.5 --- .../traceai_pi_coding_agent/README.md | 14 ++-- .../contract/README.md | 3 +- .../src/__tests__/fixtures/events.ts | 55 +++++++++++- .../src/__tests__/sessionTracer.test.ts | 84 +++++++++++++++---- .../src/sessionTracer.ts | 22 +++-- 5 files changed, 147 insertions(+), 31 deletions(-) diff --git a/typescript/packages/traceai_pi_coding_agent/README.md b/typescript/packages/traceai_pi_coding_agent/README.md index 47bcc92a2..c2eb03206 100644 --- a/typescript/packages/traceai_pi_coding_agent/README.md +++ b/typescript/packages/traceai_pi_coding_agent/README.md @@ -9,7 +9,7 @@ This is the in-process SDK integration. It is not the Pi command-line app (`pi`, | Item | Value | |---|---| | Tested SDK | `@earendil-works/pi-coding-agent` 1.2.0 (Node 22.19+) | -| Supported Pi versions | **0.80.5 and later (including 1.x):** a run ends at `agent_settled`, after Pi's retries, compaction and continuations. 0.80.5 is the first published version with `agent_settled`. **0.75.4 to 0.80.3:** Pi had no `agent_settled`, so a run ends at the first `agent_end` whose `willRetry` is not `true`. Automatic retries stay in the run; if Pi continues after that `agent_end` (overflow-compaction recovery, messages queued by `agent_end` handlers), the continuation becomes a second trace, and a compaction Pi runs after that `agent_end` is its own one-span trace. **Before 0.75.4 (including every `@mariozechner/pi-coding-agent` release):** `agent_end` has no `willRetry`, so each automatic retry also becomes its own trace. | +| Supported Pi versions | **0.80.5 and later (including 1.x):** a run ends at `agent_settled`, after Pi's retries, compaction and continuations. 0.80.5 is the first published version with `agent_settled`. **0.75.4 to 0.80.3:** Pi had no `agent_settled`, so a run ends at the first `agent_end` whose `willRetry` is not `true`, or when an automatic retry is cancelled during its backoff (`auto_retry_end` with `success: false`, after `session.abort()` or `abortRetry()`). Automatic retries stay in the run; if Pi continues after that `agent_end` (overflow-compaction recovery, messages queued by `agent_end` handlers), the continuation becomes a second trace, and a compaction Pi runs after that `agent_end` is its own one-span trace. **Before 0.75.4 (including every `@mariozechner/pi-coding-agent` release):** `agent_end` has no `willRetry`, so each automatic retry also becomes its own trace. | | Package scope | Pi moved from `@mariozechner/*` to `@earendil-works/*` at 0.74.0. The package reads the session object structurally and has no dependency on the SDK. | | SDK surface used | `AgentSession.sessionId`, `AgentSession.sessionName` (only with `hideInputs: false`), `AgentSession.subscribe()`, `AgentSession.dispose()`, and whether `AgentSession.waitForIdle` exists (never called: Pi added it together with `agent_settled`, so its presence means runs end at `agent_settled`) | @@ -57,7 +57,7 @@ One trace per agent run (one `session.prompt()`, including Pi's automatic retrie | Span name | `fi.span.kind` / `gen_ai.span.kind` | From | Parent | |---|---|---|---| -| `invoke_agent pi` | `AGENT` | `agent_start` .. `agent_settled` (Pi before 0.80.5: .. the first `agent_end` without `willRetry: true`) | the active span at `agent_start` when the app registered a context manager (for example `AsyncLocalStorageContextManager`), else none (a new trace) | +| `invoke_agent pi` | `AGENT` | `agent_start` .. `agent_settled` (Pi before 0.80.5: .. the first `agent_end` without `willRetry: true`, or a cancelled retry's `auto_retry_end`) | the active span at `agent_start` when the app registered a context manager (for example `AsyncLocalStorageContextManager`), else none (a new trace) | | `pi.turn` | `CHAIN` | `turn_start` .. `turn_end` | run | | `chat ` | `LLM` | assistant `message_start` .. `message_end` (start/end from the message's `timestamp` + `durationMs`) | turn | | `execute_tool ` | `TOOL` | `tool_execution_start` .. `tool_execution_end` | turn, or the calling tool when the event has `parentToolCallId` | @@ -74,23 +74,23 @@ One trace per agent run (one `session.prompt()`, including Pi's automatic retrie | Run totals | `pi.agent.attempts` (agent loops in the run: the first, each automatic retry, and each continuation such as overflow-compaction recovery or queued messages), `pi.agent.retry_count` (automatic retries), `pi.agent.turn_count`, `llm_call_count`, `tool_call_count`, `pi.agent.usage.*`, `pi.agent.cost.total` | summed over the run's LLM and compaction spans. Not the promoted `gen_ai.usage.*` keys, so tokens are not counted twice | | Compaction | `pi.compaction.reason` (`manual`, `threshold`, `overflow`), `pi.compaction.tokens_before`, `pi.compaction.estimated_tokens_after`, `pi.compaction.will_retry`, and the summarization call's `gen_ai.usage.*` / `gen_ai.cost.total` | `compaction_end.result` | | Retries | span events `pi.auto_retry.start` / `.end` on the run span | session events | -| Errors | `error.type` on ERROR spans: the provider's error type (for example `invalid_request_error`), else the HTTP status, else `error`; `aborted`, `tool_error`, `compaction_error` | see below | +| Errors | `error.type` on ERROR spans: the provider's error type (for example `invalid_request_error`), else the thrown error's class name (for example `TypeError`), else the HTTP status, else `error`; `aborted`, `tool_error`, `compaction_error` | see below | Errors: -- An assistant message with `stopReason` `error` or `aborted` makes its LLM span and turn ERROR. By default the status message is built only from the stop reason, an HTTP status at the start of Pi's `errorMessage`, and the provider's error `type`, for example `error: HTTP 400 invalid_request_error` or `aborted`. The provider's raw error text (truncated to 1000 characters) is the status message only with `hideOutputs: false`: gateways and providers echo request fragments, account details and local paths in it. `aborted` also sets `pi.aborted=true`. -- The run span is ERROR when the run ends aborted (`agent_settled.aborted`) or its last model call failed, with the same status message as that call. A failed call that Pi retried successfully leaves the run OK; the failed call's own span stays ERROR. +- An assistant message with `stopReason` `error` or `aborted` makes its LLM span and turn ERROR. By default the status message is built only from the stop reason, the HTTP status Pi's `errorMessage` names (a leading `4xx`/`5xx`, or `status code NNN`), and the provider's error `type` or, when there is none, the thrown error's class name (for example `TypeError`): for example `error: HTTP 400 invalid_request_error`, `error: TypeError` or `aborted`. Each part is a number or an identifier of at most 64 characters. The provider's raw error text (truncated to 1000 characters) is the status message only with `hideOutputs: false`: gateways and providers echo request fragments, account details and local paths in it. `aborted` also sets `pi.aborted=true`. +- The run span is ERROR when the run ends aborted (`agent_settled.aborted`) or its last model call failed, with the same status message as that call (on Pi before 0.80.5, a run whose retry was cancelled ends with the failed call's status). A failed call that Pi retried successfully leaves the run OK; the failed call's own span stays ERROR. - A tool result with `isError: true` makes the tool span ERROR. The status message is the tool's error text only with `hideOutputs: false`; otherwise "tool reported an error (result hidden)". - A compaction that fails is ERROR with "compaction failed" (Pi's error text only with `hideOutputs: false`); an aborted one is ERROR with "aborted". - `session.dispose()` or the stop function during a run ends every open span with `pi.incomplete=true`, children before their parents. -Not available from the SDK's session events, so not recorded: the provider request payload (the full transcript sent on each model call), streaming deltas and time to first token, a separate per-request HTTP status attribute (only what Pi's error text starts with), the model that wrote a compaction summary, and the system prompt. +Not available from the SDK's session events, so not recorded: the provider request payload (the full transcript sent on each model call), streaming deltas and time to first token, a separate per-request HTTP status attribute (only the status Pi's error text names, in the status message), the model that wrote a compaction summary, and the system prompt. ## Privacy: content is off by default Prompts, tool arguments, tool results and assistant text of a coding agent contain source code, file contents and command output. They are **not** recorded unless you opt in. -Always exported: session id, model, provider and API, token counts and cost, stop and finish reasons, tool names and call ids, durations, turn/call counts, compaction reason and token counts, and error statuses built from the stop reason, HTTP status and provider error type. The Pi working directory is never added as an attribute. +Always exported: session id, model, provider and API, token counts and cost, stop and finish reasons, tool names and call ids, durations, turn/call counts, compaction reason and token counts, and error statuses built from the stop reason, HTTP status and provider error type or error class name. The Pi working directory is never added as an attribute. To opt in: diff --git a/typescript/packages/traceai_pi_coding_agent/contract/README.md b/typescript/packages/traceai_pi_coding_agent/contract/README.md index bca5eddca..9ab56342a 100644 --- a/typescript/packages/traceai_pi_coding_agent/contract/README.md +++ b/typescript/packages/traceai_pi_coding_agent/contract/README.md @@ -15,7 +15,8 @@ cp /sdk_types.ts . && npx tsc --noEmit --strict --skipLibCheck \ --module NodeNext --moduleResolution NodeNext --target ES2022 sdk_types.ts # models.json in $PI_CODING_AGENT_DIR defines / -FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=http://localhost:14338 FI_PROJECT_NAME=pi-sdk-contract \ +# FI_BASE_URL only for a self-hosted Future AGI (default: https://api.futureagi.com) +FI_API_KEY=... FI_SECRET_KEY=... FI_BASE_URL=http://localhost:4318 FI_PROJECT_NAME=pi-sdk-contract \ PI_CODING_AGENT_DIR=/tmp/pi-agent PI_E2E_CWD=/tmp/pi-project \ node /real_sdk_session.mjs "List the files here and read README.md" ``` diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts index e6783ab75..e1b92a809 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/fixtures/events.ts @@ -2,7 +2,7 @@ * Event sequences in the order `@earendil-works/pi-coding-agent` 1.2.0 * emits them from `AgentSession.subscribe()`. * - * `happyPath` mirrors a recorded real run (gpt-6-luna through an + * `happyPath` mirrors a recorded real run (a model behind an * OpenAI-compatible endpoint, prompt "list the files, read README.md"): same * event types, order, roles, message keys, stop reasons and usage shape. Text, * ids and token numbers are fixtures; the *_SECRET strings let tests prove @@ -48,8 +48,8 @@ function assistant( role: "assistant", content, api: "openai-completions", - provider: "cliproxy", - model: "gpt-6-luna", + provider: "my-gateway", + model: "example-model", responseId: `chatcmpl-${timestamp}`, timestamp, usage: usage(0, 0), @@ -197,6 +197,55 @@ export function legacyRetry(): PiSessionEvent[] { return retryAndToolError().filter((event) => event.type !== "agent_settled"); } +/** + * A provider error whose automatic retry is cancelled during the backoff + * (`session.abort()` or `session.abortRetry()`), in the order Pi 0.75.4 to + * 0.80.3 emit it: agent_end announces the retry, then `auto_retry_end` with + * `success: false` and no further `agent_start`. Pi 0.80.5+ then also emits + * `agent_settled`. + */ +export function cancelledRetry(): PiSessionEvent[] { + const failedStreaming = assistant(T0 + 100, [], "pending"); + const failed = assistant(T0 + 100, [], "error", { + usage: usage(0, 0), + durationMs: 300, + errorMessage: '503 {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}', + }); + return [ + { type: "agent_start" }, + { type: "turn_start" }, + ...promptMessages(), + { type: "message_start", message: failedStreaming }, + { type: "message_end", message: failed }, + { type: "turn_end", message: failed, toolResults: [] }, + { type: "agent_end", messages: [], willRetry: true }, + { type: "auto_retry_start", attempt: 1, maxAttempts: 3, delayMs: 3000, errorMessage: "503 overloaded" }, + { type: "auto_retry_end", success: false, attempt: 1, finalError: "Retry cancelled" }, + ]; +} + +/** + * Legacy Pi when every retry fails: the last agent_end has `willRetry: false`, + * then `auto_retry_end` with `success: false` reports the final failure. + */ +export function legacyRetriesExhausted(): PiSessionEvent[] { + const events = cancelledRetry(); + const finalEnd = events.pop()!; + const failed = events.find( + (event) => event.type === "message_end" && (event.message as { role?: string }).role === "assistant", + )!.message; + return [ + ...events, + { type: "agent_start" }, + { type: "turn_start" }, + { type: "message_start", message: failed }, + { type: "message_end", message: failed }, + { type: "turn_end", message: failed, toolResults: [] }, + { type: "agent_end", messages: [], willRetry: false }, + { ...finalEnd, finalError: "503 overloaded" }, + ]; +} + /** * Pi 0.80.5+: after agent_end, a message queued by an agent_end handler makes * Pi continue the same run (a second agent_start) before agent_settled. diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts index 393f85a57..c222290cd 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts @@ -19,11 +19,13 @@ import { TOOL_ERROR_SECRET, TOOL_PATH_ERROR, aborted, + cancelledRetry, compactionInRun, continuedRun, happyPath, interrupted, legacyPrompt, + legacyRetriesExhausted, legacyRetry, manualCompaction, nestedTool, @@ -53,9 +55,9 @@ describe("span tree", () => { expect(spans.map((s) => s.name).sort()).toEqual( [ SpanNames.AGENT_RUN, - "chat gpt-6-luna", - "chat gpt-6-luna", - "chat gpt-6-luna", + "chat example-model", + "chat example-model", + "chat example-model", "execute_tool ls", "execute_tool read", "pi.turn", @@ -74,7 +76,7 @@ describe("span tree", () => { const ls = one(spans, "execute_tool ls"); const firstTurn = turns.find((t) => t.attributes[Attr.PI_TURN_INDEX] === 0)!; expect(parentId(ls)).toBe(firstTurn.spanContext().spanId); - for (const llm of byName(spans, "chat gpt-6-luna")) { + for (const llm of byName(spans, "chat example-model")) { expect(turns.map((t) => t.spanContext().spanId)).toContain(parentId(llm)); } }); @@ -84,7 +86,7 @@ describe("span tree", () => { const kind = (name: string) => byName(spans, name).map((s) => [s.attributes[Attr.FI_SPAN_KIND], s.attributes[Attr.GEN_AI_SPAN_KIND]]); expect(kind(SpanNames.AGENT_RUN)).toEqual([["AGENT", "AGENT"]]); expect(new Set(kind("pi.turn").flat())).toEqual(new Set(["CHAIN"])); - expect(new Set(kind("chat gpt-6-luna").flat())).toEqual(new Set(["LLM"])); + expect(new Set(kind("chat example-model").flat())).toEqual(new Set(["LLM"])); expect(kind("execute_tool ls")).toEqual([["TOOL", "TOOL"]]); for (const span of spans) { expect(span.attributes[Attr.SESSION_ID]).toBe(SESSION_ID); @@ -101,13 +103,13 @@ describe("span tree", () => { it("records model, provider, tokens, cost and timing on LLM spans", () => { const { spans } = run(happyPath()); - const llms = byName(spans, "chat gpt-6-luna").sort((a, b) => hrToMs(a.startTime) - hrToMs(b.startTime)); + const llms = byName(spans, "chat example-model").sort((a, b) => hrToMs(a.startTime) - hrToMs(b.startTime)); const [first, second, last] = llms; expect(first.attributes).toMatchObject({ [Attr.GEN_AI_OPERATION_NAME]: "chat", - [Attr.REQUEST_MODEL]: "gpt-6-luna", - [Attr.RESPONSE_MODEL]: "gpt-6-luna", - [Attr.PROVIDER_NAME]: "cliproxy", + [Attr.REQUEST_MODEL]: "example-model", + [Attr.RESPONSE_MODEL]: "example-model", + [Attr.PROVIDER_NAME]: "my-gateway", [Attr.PI_API]: "openai-completions", [Attr.INPUT_TOKENS]: 1182, [Attr.OUTPUT_TOKENS]: 75, @@ -213,7 +215,7 @@ describe("errors", () => { // The run recovered, so the run itself is OK. expect(agent.status.code).toBe(SpanStatusCode.OK); - const failedLlm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + const failedLlm = byName(spans, "chat example-model").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; // The provider's error text is content: by default only the HTTP status leaves the process. expect(failedLlm.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 429" }); expect(failedLlm.attributes[Attr.ERROR_TYPE]).toBe("429"); @@ -231,7 +233,7 @@ describe("errors", () => { it("puts the tool and provider error text in the status only when outputs are on", () => { const { spans } = run(retryAndToolError(), CONTENT_ON); expect(one(spans, "execute_tool read").status.message).toBe(TOOL_ERROR_SECRET); - const failedLlm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + const failedLlm = byName(spans, "chat example-model").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; expect(failedLlm.status.message).toBe("429 Too Many Requests: rate limited"); }); @@ -243,7 +245,7 @@ describe("errors", () => { expect(text).not.toContain("PROVIDER_ERROR_SECRET"); expect(text).not.toContain("TOOL_PATH_SECRET"); // Status: stop reason, HTTP status and the provider's error type only. - const llm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; + const llm = byName(spans, "chat example-model").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 400 invalid_request_error" }); expect(llm.attributes[Attr.ERROR_TYPE]).toBe("invalid_request_error"); const turn = byName(spans, "pi.turn").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; @@ -268,7 +270,7 @@ describe("errors", () => { const agent = one(spans, SpanNames.AGENT_RUN); expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); expect(agent.attributes[Attr.PI_ABORTED]).toBe(true); - const llm = one(spans, "chat gpt-6-luna"); + const llm = one(spans, "chat example-model"); expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); expect(llm.attributes[Attr.PI_ABORTED]).toBe(true); expect(llm.attributes[Attr.ERROR_TYPE]).toBe("aborted"); @@ -350,6 +352,48 @@ describe("Pi versions without agent_settled (before 0.80.5)", () => { expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); }); + it("ends the run when an automatic retry is cancelled during its backoff, so the next prompt is its own trace", () => { + const { provider, exporter } = memoryProvider(); + const session = new LegacySession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + session.emit(cancelledRetry()); // session.abort() / abortRetry() during the retry sleep + // Ended before the next prompt and without dispose: complete, ERROR with the failed call's status. + const failed = one(exporter.getFinishedSpans(), SpanNames.AGENT_RUN); + expect(failed.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 503 overloaded_error" }); + expect(failed.attributes[Attr.ERROR_TYPE]).toBe("overloaded_error"); + expect(failed.attributes[Attr.PI_INCOMPLETE]).toBeUndefined(); + expect(failed.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(1); + expect(failed.attributes[Attr.PI_RUN_RETRY_COUNT]).toBe(1); + expect(failed.events.map((e) => e.name)).toEqual(["pi.auto_retry.start", "pi.auto_retry.end"]); + expect(failed.events[1].attributes).toMatchObject({ "pi.attempt": 1, "pi.success": false }); + + session.emit(legacyPrompt()); + const spans = exporter.getFinishedSpans(); + const agents = byName(spans, SpanNames.AGENT_RUN); + expect(agents).toHaveLength(2); + const [first, second] = agents.map((a) => a.spanContext().traceId); + expect(second).not.toBe(first); + expect(agents[1].status.code).toBe(SpanStatusCode.OK); + expect(agents[1].attributes[Attr.PI_RUN_ATTEMPTS]).toBe(1); + expect(agents[1].attributes[Attr.PI_RUN_RETRY_COUNT]).toBe(0); + expect(agents[1].attributes[Attr.PI_RUN_LLM_CALL_COUNT]).toBe(3); + // The failed run's trace holds only its own run, turn and model call. + expect(spans.filter((s) => s.spanContext().traceId === first).map((s) => s.name).sort()).toEqual( + [SpanNames.AGENT_RUN, "chat example-model", "pi.turn"].sort(), + ); + expect(spans.filter((s) => s.spanContext().traceId === second)).toHaveLength(9); + }); + + it("ends a run whose retries all fail at its last agent_end; the final auto_retry_end changes nothing", () => { + const { spans } = run(legacyRetriesExhausted(), {}, new LegacySession()); + const agent = one(spans, SpanNames.AGENT_RUN); + expect(agent.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(2); + expect(agent.attributes[Attr.PI_RUN_RETRY_COUNT]).toBe(1); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 503 overloaded_error" }); + expect(spans).toHaveLength(5); // run, 2 turns, 2 model calls + expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); + }); + it("switches to agent_settled once a session without waitForIdle() is seen to settle", () => { const { provider, exporter } = memoryProvider(); const session = new LegacySession(); @@ -369,6 +413,18 @@ describe("Pi versions without agent_settled (before 0.80.5)", () => { expect(agent.attributes[Attr.PI_RUN_TURN_COUNT]).toBe(4); expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); }); + + it("leaves a cancelled retry to agent_settled when the session has waitForIdle() (Pi 0.80.5+)", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider }).instrumentSession(session); + session.emit(cancelledRetry()); + expect(byName(exporter.getFinishedSpans(), SpanNames.AGENT_RUN)).toHaveLength(0); + session.emit([{ type: "agent_settled", aborted: true }]); + const agent = one(exporter.getFinishedSpans(), SpanNames.AGENT_RUN); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); + expect(agent.events.map((e) => e.name)).toEqual(["pi.auto_retry.start", "pi.auto_retry.end"]); + }); }); describe("compaction", () => { @@ -462,7 +518,7 @@ describe("content policy", () => { const ls = one(spans, "execute_tool ls"); expect(JSON.parse(ls.attributes[Attr.INPUT_VALUE] as string)).toEqual({ path: ARG_SECRET }); expect(ls.attributes[Attr.OUTPUT_VALUE]).toBe(RESULT_SECRET); - const firstLlm = byName(spans, "chat gpt-6-luna").find((s) => s.attributes[Attr.PI_TOOL_CALL_COUNT] === 1)!; + const firstLlm = byName(spans, "chat example-model").find((s) => s.attributes[Attr.PI_TOOL_CALL_COUNT] === 1)!; expect(firstLlm.attributes[Attr.OUTPUT_VALUE]).toContain("call_ls_1"); // The system prompt is never exported, even with content on. expect(exportedText(spans)).not.toContain(SYSTEM_SECRET); diff --git a/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts index d90914411..bf8d9fa3d 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts @@ -18,9 +18,10 @@ * Run end: Pi 0.80.5 and later emit `agent_settled` once Pi will not continue * on its own; the run ends there. Older Pi has no `agent_settled`, so until * the session is known to settle, the run ends at an `agent_end` that does not - * announce a retry (`willRetry !== true`). Pi added `AgentSession.waitForIdle()` - * in the same release as `agent_settled`, so a session that has it is treated - * as settling from the start. + * announce a retry (`willRetry !== true`), or at an `auto_retry_end` with + * `success: false` (a retry cancelled during its backoff). Pi added + * `AgentSession.waitForIdle()` in the same release as `agent_settled`, so a + * session that has it is treated as settling from the start. * * Every span carries `session.id` = `session.sessionId`, so all runs of one * session group into one Future AGI session. Listeners must never throw into @@ -99,9 +100,12 @@ const IDENTIFIER = /^[A-Za-z][A-Za-z0-9_.-]{0,63}$/; * Pi's `errorMessage` is the provider's raw error text. Gateways and * providers echo request fragments, account details and local paths in it, * so it is content: it is the status message only when outputs are shown. - * Otherwise the message is built from the stop reason, the HTTP status at the - * start of the text and the provider's error `type` (an identifier such as - * `invalid_request_error`), for example `error: HTTP 400 invalid_request_error`. + * Otherwise the message is built from the stop reason, the HTTP status the + * text names (a leading `4xx`/`5xx`, or `status code NNN`) and the provider's + * error `type` (an identifier such as `invalid_request_error`) or, without + * one, the thrown error's class name from `diagnostics` (such as `TypeError`), + * for example `error: HTTP 400 invalid_request_error`. All are numbers or + * identifiers of at most 64 characters. */ export function errorDetails( message: PiMessage | undefined, @@ -248,6 +252,12 @@ export class PiSessionTracer { "pi.attempt": num(event.attempt), "pi.success": event.success === true, }); + // Older Pi: abort() or abortRetry() during the retry backoff ends the + // retry with success:false and no further agent_start, after an + // agent_end that announced the retry. End the run here, or the next + // prompt would join it. (When retries run out, the run already ended + // at the agent_end without willRetry, so this is a no-op.) + if (!this.settles && event.success === false) this.endRun(false); break; case "compaction_start": this.onCompactionStart(event); From b10e1c9dafbb8e53175d8de9a920dac5208a8f1c Mon Sep 17 00:00:00 2001 From: Nikhil Pareek <8347993+nik13@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:04:54 +0000 Subject: [PATCH 4/7] ci: build and test TypeScript, Go and Java integrations Plan only supported packages present at the head. Preserve Python and LiveKit coverage; use frozen dependencies, pinned toolchains and a fail-closed aggregate. No vendor, publication or deployment credentials. Co-Authored-By: Claude Code --- .github/integration-ci/check_results.py | 22 ++ .github/integration-ci/install_maven.sh | 14 ++ .github/integration-ci/integration_plan.py | 83 +++++++ .github/integration-ci/test_check_results.py | 42 ++++ .../integration-ci/test_integration_plan.py | 109 +++++++++ .github/workflows/integration-tests.yml | 210 ++++++++++++++++++ 6 files changed, 480 insertions(+) create mode 100644 .github/integration-ci/check_results.py create mode 100644 .github/integration-ci/install_maven.sh create mode 100644 .github/integration-ci/integration_plan.py create mode 100644 .github/integration-ci/test_check_results.py create mode 100644 .github/integration-ci/test_integration_plan.py create mode 100644 .github/workflows/integration-tests.yml diff --git a/.github/integration-ci/check_results.py b/.github/integration-ci/check_results.py new file mode 100644 index 000000000..1ea3e7ee0 --- /dev/null +++ b/.github/integration-ci/check_results.py @@ -0,0 +1,22 @@ +#!/usr/bin/env python3 +"""Fail if a planned matrix was skipped, failed or cancelled (no empty green gate).""" +import json +import os + + +def check(needs): + if needs['plan']['result'] != 'success': + raise ValueError('integration planning did not succeed') + for language in ['typescript', 'go', 'java']: + matrix = json.loads(needs['plan']['outputs'][language]) + if not isinstance(matrix, list): + raise ValueError(f'{language}: missing/invalid matrix') + expected = 'success' if matrix else 'skipped' + actual = needs[language]['result'] + if actual != expected: + raise ValueError(f'{language}: planned {len(matrix)} packages, got {actual}, expected {expected}') + + +if __name__ == '__main__': + check(json.loads(os.environ['NEEDS_JSON'])) + print('Every planned integration built and passed its tests.') diff --git a/.github/integration-ci/install_maven.sh b/.github/integration-ci/install_maven.sh new file mode 100644 index 000000000..7285dfa00 --- /dev/null +++ b/.github/integration-ci/install_maven.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Exact Maven toolchain, verified before extracting; no third-party setup action. +set -euo pipefail +: "${RUNNER_TEMP:?}" +: "${GITHUB_PATH:?}" +archive="$RUNNER_TEMP/apache-maven-3.9.9-bin.tar.gz" +curl --fail --show-error --location --retry 3 --max-time 120 \ + https://repo.maven.apache.org/maven2/org/apache/maven/apache-maven/3.9.9/apache-maven-3.9.9-bin.tar.gz \ + --output "$archive" +printf '%s %s\n' \ + a555254d6b53d267965a3404ecb14e53c3827c09c3b94b5678835887ab404556bfaf78dcfe03ba76fa2508649dca8531c74bca4d5846513522404d48e8c4ac8b \ + "$archive" | sha512sum --check --strict +tar -xzf "$archive" -C "$RUNNER_TEMP" +printf '%s\n' "$RUNNER_TEMP/apache-maven-3.9.9/bin" >> "$GITHUB_PATH" diff --git a/.github/integration-ci/integration_plan.py b/.github/integration-ci/integration_plan.py new file mode 100644 index 000000000..918a384ea --- /dev/null +++ b/.github/integration-ci/integration_plan.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +"""Plan the Codex/Pi, Go and Java integration checks, without changing Python CI. + +Uses the existing Python planner's NUL-safe, rename-aware git diff reader. The +same files can land independently with each integration: absent target directories +are ignored, but an existing target may not silently lose its manifest or tests. +CI changes run every present target; TS shared dependencies run both TS targets. +""" +import argparse +import json +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / '.github/scripts')) +from python_test_plan import PlanError, changed_paths, commit_count + +TARGETS = ( + ('codex-sdk', 'typescript', 'typescript/packages/traceai_codex_sdk', 'package.json'), + ('pi-coding-agent', 'typescript', 'typescript/packages/traceai_pi_coding_agent', 'package.json'), + ('openai', 'go', 'go/openai', 'go.mod'), + ('a2a', 'go', 'go/a2a', 'go.mod'), + ('eino', 'go', 'go/eino', 'go.mod'), + ('arconia', 'java', 'java/examples/arconia-example', 'pom.xml'), + ('quarkus-langchain4j', 'java', 'java/examples/quarkus-langchain4j-example', 'pom.xml'), +) +WORKFLOW = '.github/workflows/integration-tests.yml' +CI_PREFIX = '.github/integration-ci/' +SHARED_DIFF_READER = '.github/scripts/python_test_plan.py' +TS_SHARED_FILES = { + 'typescript/package.json', 'typescript/pnpm-lock.yaml', + 'typescript/pnpm-workspace.yaml', 'typescript/.npmrc', +} +TS_SHARED_PREFIXES = ('typescript/packages/fi-core/', 'typescript/packages/fi-semantic-conventions/') +TEST_GLOBS = {'typescript': 'src/__tests__/**/*.test.ts', 'go': '**/*_test.go', 'java': 'src/test/java/**/*Test.java'} + + +def plan(changed, root=ROOT, all_targets=False): + changed = set(changed) + run_all = all_targets or WORKFLOW in changed or SHARED_DIFF_READER in changed or any(p.startswith(CI_PREFIX) for p in changed) + ts_shared = any(p in TS_SHARED_FILES or p.startswith(TS_SHARED_PREFIXES) or + (p.startswith('typescript/tsconfig') and p.endswith('.json')) for p in changed) + result = {'typescript': [], 'go': [], 'java': []} + for name, language, directory, manifest in TARGETS: + path = root / directory + if not path.is_dir(): + # A not-yet-landed or entirely deleted integration is not a target. + continue + if not (path / manifest).is_file(): + raise PlanError(f'{directory} exists but is missing {manifest}') + if not any(path.glob(TEST_GLOBS[language])): + raise PlanError(f'{directory} has no tests; refusing an empty test gate') + if run_all or (language == 'typescript' and ts_shared) or any(p.startswith(directory + '/') for p in changed): + result[language].append({'name': name, 'dir': directory}) + return result + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + args = parser.add_mutually_exclusive_group() + args.add_argument('--diff') + args.add_argument('--all', action='store_true') + options = parser.parse_args(argv) + try: + if options.diff: + paths = changed_paths(options.diff, ROOT) + if not paths and commit_count(options.diff, ROOT) == 0: + raise PlanError('no commits in diff; check base/head wiring') + elif options.all: + paths = changed_paths(None, ROOT) + if not paths: + raise PlanError('no tracked files; check checkout') + else: + paths = [line.strip() for line in sys.stdin if line.strip()] + print(json.dumps(plan(paths, all_targets=options.all), sort_keys=True)) + except PlanError as exc: + print(f'integration_plan: {exc}', file=sys.stderr) + return 1 + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/.github/integration-ci/test_check_results.py b/.github/integration-ci/test_check_results.py new file mode 100644 index 000000000..349c9099f --- /dev/null +++ b/.github/integration-ci/test_check_results.py @@ -0,0 +1,42 @@ +import copy +import unittest +from check_results import check + + +class ResultsTests(unittest.TestCase): + def setUp(self): + self.needs = {'plan': {'result': 'success', 'outputs': { + 'typescript': '[]', 'go': '[{"name":"openai"}]', 'java': '[]'}}, + 'typescript': {'result': 'skipped'}, 'go': {'result': 'success'}, + 'java': {'result': 'skipped'}} + + def test_success(self): + check(self.needs) + + def test_failed_or_skipped_planner_fails(self): + for result in ['failure', 'cancelled', 'skipped']: + needs = copy.deepcopy(self.needs) + needs['plan']['result'] = result + with self.assertRaises(ValueError): + check(needs) + + def test_planned_matrix_must_succeed(self): + for result in ['failure', 'cancelled', 'skipped']: + needs = copy.deepcopy(self.needs) + needs['go']['result'] = result + with self.assertRaises(ValueError): + check(needs) + + def test_unplanned_matrix_must_not_claim_success(self): + self.needs['java']['result'] = 'success' + with self.assertRaises(ValueError): + check(self.needs) + + def test_missing_or_invalid_matrix_fails(self): + for value in ['', 'null', '{}', 'false']: + self.needs['plan']['outputs']['go'] = value + with self.assertRaises(ValueError): + check(self.needs) + del self.needs['plan']['outputs']['go'] + with self.assertRaises(KeyError): + check(self.needs) diff --git a/.github/integration-ci/test_integration_plan.py b/.github/integration-ci/test_integration_plan.py new file mode 100644 index 000000000..91c6e5c3b --- /dev/null +++ b/.github/integration-ci/test_integration_plan.py @@ -0,0 +1,109 @@ +"""Planner regression checks: every new path, shared dependencies and safe skips.""" +import shutil +import subprocess +import tempfile +import unittest +from pathlib import Path + +from integration_plan import TARGETS, WORKFLOW, PlanError, plan, changed_paths + + +class IntegrationPlanTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.tests = {} + for name, language, directory, manifest in TARGETS: + package = self.root / directory + package.mkdir(parents=True) + (package / manifest).write_text('{}') + test = {'typescript': 'src/__tests__/tracer.test.ts', 'go': 'tracer_test.go', + 'java': 'src/test/java/example/ExportTest.java'}[language] + test_path = package / test + test_path.parent.mkdir(parents=True, exist_ok=True) + test_path.touch() + self.tests[name] = test_path + + def names(self, paths, **kwargs): + return {p['name'] for matrix in plan(paths, self.root, **kwargs).values() for p in matrix} + + def test_every_owned_path_selects_its_own_target(self): + for name, _, directory, manifest in TARGETS: + for suffix in [manifest, 'README.md', 'example/main.go', 'src/changed.ts']: + with self.subTest(name=name, suffix=suffix): + self.assertEqual(self.names([directory + '/' + suffix]), {name}) + + def test_ci_edits_and_all_select_every_present_target(self): + expected = {t[0] for t in TARGETS} + for path in [WORKFLOW, '.github/integration-ci/integration_plan.py', + '.github/integration-ci/test_integration_plan.py', + '.github/integration-ci/install_maven.sh', '.github/scripts/python_test_plan.py']: + self.assertEqual(self.names([path]), expected) + self.assertEqual(self.names([], all_targets=True), expected) + + def test_shared_typescript_inputs_cover_both_consumers(self): + for path in ['typescript/pnpm-lock.yaml', 'typescript/package.json', + 'typescript/pnpm-workspace.yaml', 'typescript/.npmrc', + 'typescript/tsconfig.base.json', 'typescript/packages/fi-core/src/index.ts', + 'typescript/packages/fi-semantic-conventions/src/index.ts']: + with self.subTest(path=path): + self.assertEqual(self.names([path]), {'codex-sdk', 'pi-coding-agent'}) + + def test_unrelated_paths_do_not_claim_coverage(self): + self.assertEqual(self.names(['README.md', 'python/frameworks/openai/tracer.py', + '.github/workflows/python-tests.yml', + 'typescript/packages/traceai_openai/src/index.ts', + 'java/traceai-core/pom.xml', 'go/openai-other/main.go']), set()) + + def test_target_not_yet_landed_is_ignored(self): + for _, _, directory, _ in TARGETS[1:]: + shutil.rmtree(self.root / directory) + self.assertEqual(self.names([WORKFLOW]), {'codex-sdk'}) + self.assertEqual(self.names(['go/a2a/interceptor.go']), set()) + + def test_entirely_deleted_target_is_ignored(self): + shutil.rmtree(self.root / 'go/openai') + self.assertEqual(self.names(['go/openai/go.mod']), set()) + + def test_missing_manifest_fails_closed(self): + (self.root / 'go/openai/go.mod').unlink() + with self.assertRaisesRegex(PlanError, 'missing go.mod'): + plan(['go/openai/tracer.go'], self.root) + + def test_no_tests_fails_closed_for_each_language(self): + for name in ['codex-sdk', 'openai', 'arconia']: + with self.subTest(name=name): + self.tests[name].unlink() + with self.assertRaisesRegex(PlanError, 'no tests'): + plan([WORKFLOW], self.root) + self.tests[name].touch() + + def test_duplicate_paths_do_not_duplicate_jobs(self): + result = plan(['go/openai/go.mod'] * 5, self.root) + self.assertEqual(len(result['go']), 1) + + def test_moves_plan_both_old_and_new_owner_using_existing_diff_reader(self): + # Exercise real git's --no-renames/-z path handling, not just an artificial list. + def git(*args): + return subprocess.check_output(['git', '-C', str(self.root), *args], stderr=subprocess.STDOUT) + git('init', '-q') + git('add', '.') + env = __import__('os').environ.copy() + env.update(GIT_AUTHOR_NAME='CI test', GIT_COMMITTER_NAME='CI test', + GIT_AUTHOR_EMAIL='ci@example.invalid', GIT_COMMITTER_EMAIL='ci@example.invalid') + subprocess.run(['git', '-C', str(self.root), 'commit', '-qm', 'fixture'], env=env, check=True) + old = self.root / 'go/openai/with tab\tname.go' + old.touch() + git('add', '.') + subprocess.run(['git', '-C', str(self.root), 'commit', '-qm', 'old owner'], env=env, check=True) + old.rename(self.root / 'go/a2a/with tab\tname.go') + git('add', '.') + subprocess.run(['git', '-C', str(self.root), 'commit', '-qm', 'move'], env=env, check=True) + paths = changed_paths('HEAD~1..HEAD', self.root) + self.assertIn('go/openai/with tab\tname.go', paths) + self.assertEqual(self.names(paths), {'openai', 'a2a'}) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml new file mode 100644 index 000000000..7546c0451 --- /dev/null +++ b/.github/workflows/integration-tests.yml @@ -0,0 +1,210 @@ +name: Integration tests + +# Additive to python-tests.yml and livekit-release-check.yml. Only the seven +# targets listed in integration_plan.py are covered here; absent, not-yet-merged +# packages are ignored so each integration can carry this same CI independently. +on: + pull_request: + branches: [main, dev] + types: [opened, synchronize, reopened, edited] + push: + branches: [main, dev] + workflow_dispatch: + +permissions: + contents: read + +defaults: + run: + shell: bash + +concurrency: + group: integration-tests-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +env: + FI_BASE_URL: http://127.0.0.1:9 + FI_GRPC_URL: http://127.0.0.1:9 + # Contract tests use local stand-ins; this workflow never needs vendor secrets. + CI: 'true' + +jobs: + plan: + name: Integration plan + runs-on: ubuntu-24.04 + timeout-minutes: 10 + outputs: + typescript: ${{ steps.plan.outputs.typescript }} + go: ${{ steps.plan.outputs.go }} + java: ${{ steps.plan.outputs.java }} + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + ref: ${{ github.event.pull_request.head.sha || github.sha }} + fetch-depth: 0 + persist-credentials: false + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: '3.12.3' + - name: Test existing and integration planners and result gates + run: | + python -m unittest discover -s .github/scripts -p 'test_*.py' -v + python -m unittest discover -s .github/integration-ci -p 'test_*.py' -v + - name: Plan changed integrations + id: plan + env: + EVENT_NAME: ${{ github.event_name }} + PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} + PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + PUSH_BEFORE_SHA: ${{ github.event.before }} + PUSH_AFTER_SHA: ${{ github.event.after }} + run: | + planner=.github/integration-ci/integration_plan.py + plan_file="$RUNNER_TEMP/integration-plan.json" + case "$EVENT_NAME" in + pull_request) + if ! git cat-file -e "$PR_BASE_SHA^{commit}" 2>/dev/null; then + timeout 120 git fetch --no-tags origin "$PR_BASE_SHA" || true + fi + if git cat-file -e "$PR_BASE_SHA^{commit}" 2>/dev/null; then + python "$planner" --diff "$PR_BASE_SHA...$PR_HEAD_SHA" > "$plan_file" + else + python "$planner" --all > "$plan_file" + fi + ;; + push) + if [ "$PUSH_BEFORE_SHA" = 0000000000000000000000000000000000000000 ] || ! git cat-file -e "$PUSH_BEFORE_SHA^{commit}" 2>/dev/null; then + python "$planner" --all > "$plan_file" + else + python "$planner" --diff "$PUSH_BEFORE_SHA..$PUSH_AFTER_SHA" > "$plan_file" + fi + ;; + *) python "$planner" --all > "$plan_file" ;; + esac + jq . "$plan_file" + for language in typescript go java; do + echo "$language=$(jq -c --arg language "$language" '.[$language]' "$plan_file")" >> "$GITHUB_OUTPUT" + done + + typescript: + name: TypeScript (${{ matrix.package.name }}) + needs: plan + if: needs.plan.outputs.typescript != '[]' + runs-on: ubuntu-24.04 + timeout-minutes: 25 + strategy: + fail-fast: false + matrix: + package: ${{ fromJSON(needs.plan.outputs.typescript) }} + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + ref: ${{ github.event.pull_request.head.sha || github.sha }} + persist-credentials: false + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version: '22.23.4' + - name: Install locked target and workspace dependencies + working-directory: typescript + env: + PACKAGE_NAME: ${{ matrix.package.name }} + run: | + # packageManager pins pnpm 10.30.1 with its integrity hash. + corepack pnpm --version + corepack pnpm --filter "@traceai/$PACKAGE_NAME..." install --frozen-lockfile --ignore-scripts + - name: Build workspace dependencies and target, then run unit tests + working-directory: typescript + env: + PACKAGE_NAME: ${{ matrix.package.name }} + run: | + corepack pnpm run build:deps + corepack pnpm --filter "@traceai/$PACKAGE_NAME" run build + corepack pnpm --filter "@traceai/$PACKAGE_NAME" test --runInBand + - name: Codex real SDK contract (local subprocess, no API calls) + if: matrix.package.name == 'codex-sdk' + working-directory: ${{ matrix.package.dir }} + run: node contract/run_real_sdk.mjs + - name: Build installable tarball + working-directory: ${{ matrix.package.dir }} + run: | + mkdir -p "$RUNNER_TEMP/package" + corepack pnpm pack --pack-destination "$RUNNER_TEMP/package" + test -n "$(find "$RUNNER_TEMP/package" -name '*.tgz' -print -quit)" + git diff --exit-code -- ../../pnpm-lock.yaml + + go: + name: Go (${{ matrix.package.name }}) + needs: plan + if: needs.plan.outputs.go != '[]' + runs-on: ubuntu-24.04 + timeout-minutes: 25 + strategy: + fail-fast: false + matrix: + package: ${{ fromJSON(needs.plan.outputs.go) }} + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + ref: ${{ github.event.pull_request.head.sha || github.sha }} + persist-credentials: false + - uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5.5.0 + with: + go-version: '1.26.0' + cache-dependency-path: ${{ matrix.package.dir }}/go.sum + - name: Verify modules, build package and example, vet and race-test + working-directory: ${{ matrix.package.dir }} + env: + GOTOOLCHAIN: local + GOFLAGS: -mod=readonly + run: | + go mod download + go mod verify + go build ./... + go vet ./... + go test -race -count=1 -v ./... + git diff --exit-code -- go.mod go.sum + + java: + name: Java (${{ matrix.package.name }}) + needs: plan + if: needs.plan.outputs.java != '[]' + runs-on: ubuntu-24.04 + timeout-minutes: 30 + strategy: + fail-fast: false + matrix: + package: ${{ fromJSON(needs.plan.outputs.java) }} + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + ref: ${{ github.event.pull_request.head.sha || github.sha }} + persist-credentials: false + - uses: actions/setup-java@c5195efecf7bdfc987ee8bae7a71cb8b11521c00 # v4.7.1 + with: + distribution: temurin + java-version: '21.0.8+9' + - name: Install checksum-pinned Maven 3.9.9 + run: bash .github/integration-ci/install_maven.sh + - name: Build packaged application and run OTLP integration tests + working-directory: ${{ matrix.package.dir }} + run: | + mvn --version + mvn -B -ntp -DfailIfNoTests=true verify + test -n "$(find target -name '*.jar' -print -quit)" + git diff --exit-code -- pom.xml + + result: + name: Integration tests + needs: [plan, typescript, go, java] + if: always() + runs-on: ubuntu-24.04 + timeout-minutes: 5 + steps: + - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + with: + ref: ${{ github.event.pull_request.head.sha || github.sha }} + persist-credentials: false + - name: Require success from every planned language matrix + env: + NEEDS_JSON: ${{ toJSON(needs) }} + run: python3 .github/integration-ci/check_results.py From e4770a8838233d78558af15e91166ae4f9e319ef Mon Sep 17 00:00:00 2001 From: Nikhil Pareek <8347993+nik13@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:49:04 +0000 Subject: [PATCH 5/7] fix: honor independent error-detail privacy after core upgrade Keep default error descriptions private even when outputs are enabled. Preserve statuses/types, require both opt-ins for raw errors, and cover custom attributes, failed tool results, retries and exception paths with regression tests. Co-Authored-By: Claude Code --- .../traceai_pi_coding_agent/README.md | 14 ++++- .../src/__tests__/errorPrivacy.test.ts | 60 +++++++++++++++++++ .../src/__tests__/sessionTracer.test.ts | 38 ++++++------ .../traceai_pi_coding_agent/src/content.ts | 2 + .../src/instrumentation.ts | 4 +- .../src/sessionTracer.ts | 22 ++++--- 6 files changed, 108 insertions(+), 32 deletions(-) create mode 100644 typescript/packages/traceai_pi_coding_agent/src/__tests__/errorPrivacy.test.ts diff --git a/typescript/packages/traceai_pi_coding_agent/README.md b/typescript/packages/traceai_pi_coding_agent/README.md index c2eb03206..dbd3c3185 100644 --- a/typescript/packages/traceai_pi_coding_agent/README.md +++ b/typescript/packages/traceai_pi_coding_agent/README.md @@ -90,7 +90,7 @@ Not available from the SDK's session events, so not recorded: the provider reque Prompts, tool arguments, tool results and assistant text of a coding agent contain source code, file contents and command output. They are **not** recorded unless you opt in. -Always exported: session id, model, provider and API, token counts and cost, stop and finish reasons, tool names and call ids, durations, turn/call counts, compaction reason and token counts, and error statuses built from the stop reason, HTTP status and provider error type or error class name. The Pi working directory is never added as an attribute. +Always exported: session id, model, provider and API, token counts and cost, stop and finish reasons, tool names and call ids, durations, turn/call counts, compaction reason and token counts, and error status codes and provider error types or error class names. The Pi working directory is never added as an attribute. To opt in: @@ -102,9 +102,17 @@ new PiCodingAgentInstrumentation({ tracerProvider, traceConfig: { hideInputs: fa | Setting | Adds | |---|---| | `hideInputs: false` | the user prompt (`input.value` on the run span), tool arguments (`input.value` on tool spans), tool-call arguments inside the LLM output, the session name (`pi.session.name`) | -| `hideOutputs: false` | assistant text (`output.value` on LLM and run spans), tool results (`output.value` on tool spans), the compaction summary (`output.value` on `pi.compaction`), and the raw error text in the status: provider errors (up to 1000 characters), tool errors, compaction errors | +| `hideOutputs: false` | assistant text (`output.value` on LLM and run spans), tool results (`output.value` on tool spans), the compaction summary (`output.value` on `pi.compaction`) | -With content on, tool arguments, tool results and error text usually contain absolute file paths, which include your username. +Error details are independently hidden by `hideErrorDetails: true` (the default) +or `FI_HIDE_ERROR_DETAILS=true`. Status codes and error types remain; status +messages, failed tool results, failed model output and compaction error text do +not. `hideOutputs: false` alone does not reveal them. To include error text, set +both `hideOutputs: false` and `hideErrorDetails: false` (or both +`FI_HIDE_OUTPUTS=false` and `FI_HIDE_ERROR_DETAILS=false`). + +With content and error details on, tool arguments, tool results and error text +usually contain absolute file paths, which include your username. Precedence: `traceConfig` option, then the env var, then hidden. The env vars fail closed: only `false` (any case, surrounding spaces ignored) turns capture on. Values are cut at `maxContentLength` characters (default 16384). The system prompt is never exported. diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/errorPrivacy.test.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/errorPrivacy.test.ts new file mode 100644 index 000000000..91a9f8fe2 --- /dev/null +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/errorPrivacy.test.ts @@ -0,0 +1,60 @@ +import { SpanStatusCode } from "@opentelemetry/api"; +import { PiCodingAgentInstrumentation } from "../index"; +import { resolveContentPolicy } from "../instrumentation"; +import { PiMessage } from "../types"; +import { FakeSession, PROVIDER_ERROR_TEXT, TOOL_PATH_ERROR, providerAndToolPathErrors } from "./fixtures/events"; +import { exportedText, memoryProvider } from "./helpers"; + +const MARKER = "FAILED_MODEL_OUTPUT_PRIVATE"; +const before = process.env.FI_HIDE_ERROR_DETAILS; +beforeEach(() => { delete process.env.FI_HIDE_ERROR_DETAILS; }); +afterEach(() => { + if (before === undefined) delete process.env.FI_HIDE_ERROR_DETAILS; + else process.env.FI_HIDE_ERROR_DETAILS = before; +}); + +it.each([undefined, true])("output opt-in alone keeps failed tool results and model errors private (%s)", (hideErrorDetails) => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider, traceConfig: { hideOutputs: false, hideErrorDetails } }).instrumentSession(session); + const events = providerAndToolPathErrors(); + for (const event of events) { + const message = event.message as PiMessage | undefined; + if (message?.stopReason === "error") message.content = [{ type: "text", text: MARKER }]; + } + session.emit(events); + const spans = exporter.getFinishedSpans(); + const failed = spans.filter((s) => s.status.code === SpanStatusCode.ERROR); + expect(failed.length).toBeGreaterThanOrEqual(4); + for (const span of failed) { + expect(span.status.message).toBeUndefined(); + expect(span.attributes["error.type"]).toBeDefined(); + } + const text = exportedText(spans); + for (const privateText of [MARKER, "PROVIDER_ERROR_SECRET", "TOOL_PATH_SECRET", "/Users/"]) { + expect(text).not.toContain(privateText); + } +}); + +it("both opt-ins retain raw tool and provider errors", () => { + const { provider, exporter } = memoryProvider(); + const session = new FakeSession(); + new PiCodingAgentInstrumentation({ tracerProvider: provider, traceConfig: { hideOutputs: false, hideErrorDetails: false } }).instrumentSession(session); + session.emit(providerAndToolPathErrors()); + const text = exportedText(exporter.getFinishedSpans()); + expect(text).toContain(JSON.stringify(PROVIDER_ERROR_TEXT).slice(1, -1)); + expect(text).toContain(JSON.stringify(TOOL_PATH_ERROR).slice(1, -1)); +}); + +it.each([undefined, "", "0", "off", "true", "invalid"])("error privacy fails closed for env %s", (value) => { + if (value !== undefined) process.env.FI_HIDE_ERROR_DETAILS = value; + expect(resolveContentPolicy().hideErrorDetails).toBe(true); +}); + +it("explicit options take precedence over error-detail environment settings", () => { + process.env.FI_HIDE_ERROR_DETAILS = " FALSE "; + expect(resolveContentPolicy().hideErrorDetails).toBe(false); + expect(resolveContentPolicy({ hideErrorDetails: true }).hideErrorDetails).toBe(true); + process.env.FI_HIDE_ERROR_DETAILS = "true"; + expect(resolveContentPolicy({ hideErrorDetails: false }).hideErrorDetails).toBe(false); +}); diff --git a/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts index c222290cd..17eea32d4 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/__tests__/sessionTracer.test.ts @@ -47,7 +47,7 @@ function run( return { spans: exporter.getFinishedSpans(), session }; } -const CONTENT_ON = { traceConfig: { hideInputs: false, hideOutputs: false } }; +const CONTENT_ON = { traceConfig: { hideInputs: false, hideOutputs: false, hideErrorDetails: false } }; describe("span tree", () => { it("builds one agent run with turns, LLM calls and tools, all in one trace", () => { @@ -216,8 +216,8 @@ describe("errors", () => { expect(agent.status.code).toBe(SpanStatusCode.OK); const failedLlm = byName(spans, "chat example-model").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; - // The provider's error text is content: by default only the HTTP status leaves the process. - expect(failedLlm.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 429" }); + // Status code and error type remain; the independent error-detail policy strips status text. + expect(failedLlm.status).toEqual({ code: SpanStatusCode.ERROR }); expect(failedLlm.attributes[Attr.ERROR_TYPE]).toBe("429"); const failedTurn = byName(spans, "pi.turn").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; expect(failedTurn.status.code).toBe(SpanStatusCode.ERROR); @@ -226,11 +226,11 @@ describe("errors", () => { expect(tool.status.code).toBe(SpanStatusCode.ERROR); expect(tool.attributes[Attr.PI_TOOL_IS_ERROR]).toBe(true); // Tool error text is tool output: hidden by default. - expect(tool.status.message).toBe(TOOL_ERROR_HIDDEN); + expect(tool.status.message).toBeUndefined(); expect(tool.attributes[Attr.ERROR_TYPE]).toBe("tool_error"); }); - it("puts the tool and provider error text in the status only when outputs are on", () => { + it("puts the tool and provider error text in the status only when outputs and error details are on", () => { const { spans } = run(retryAndToolError(), CONTENT_ON); expect(one(spans, "execute_tool read").status.message).toBe(TOOL_ERROR_SECRET); const failedLlm = byName(spans, "chat example-model").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; @@ -244,19 +244,19 @@ describe("errors", () => { expect(text).not.toContain("/Users/"); expect(text).not.toContain("PROVIDER_ERROR_SECRET"); expect(text).not.toContain("TOOL_PATH_SECRET"); - // Status: stop reason, HTTP status and the provider's error type only. + // Status code and provider error type remain, without status text. const llm = byName(spans, "chat example-model").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; - expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 400 invalid_request_error" }); + expect(llm.status).toEqual({ code: SpanStatusCode.ERROR }); expect(llm.attributes[Attr.ERROR_TYPE]).toBe("invalid_request_error"); const turn = byName(spans, "pi.turn").find((s) => s.attributes[Attr.PI_STOP_REASON] === "error")!; - expect(turn.status.message).toBe("error: HTTP 400 invalid_request_error"); + expect(turn.status.message).toBeUndefined(); const agent = one(spans, SpanNames.AGENT_RUN); - expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 400 invalid_request_error" }); - expect(one(spans, "execute_tool read").status).toEqual({ code: SpanStatusCode.ERROR, message: TOOL_ERROR_HIDDEN }); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(one(spans, "execute_tool read").status).toEqual({ code: SpanStatusCode.ERROR }); }); - it("exports the raw provider and tool error text, paths included, only with outputs on", () => { - const text = exportedText(run(providerAndToolPathErrors(), { traceConfig: { hideOutputs: false } }).spans); + it("exports the raw provider and tool error text, paths included, only with outputs and error details on", () => { + const text = exportedText(run(providerAndToolPathErrors(), { traceConfig: { hideOutputs: false, hideErrorDetails: false } }).spans); expect(text).toContain(JSON.stringify(PROVIDER_ERROR_TEXT).slice(1, -1)); expect(text).toContain(JSON.stringify(TOOL_PATH_ERROR).slice(1, -1)); // Inputs alone do not reveal error text. @@ -268,10 +268,10 @@ describe("errors", () => { it("marks an aborted run and its model call ERROR", () => { const { spans } = run(aborted()); const agent = one(spans, SpanNames.AGENT_RUN); - expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR }); expect(agent.attributes[Attr.PI_ABORTED]).toBe(true); const llm = one(spans, "chat example-model"); - expect(llm.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); + expect(llm.status).toEqual({ code: SpanStatusCode.ERROR }); expect(llm.attributes[Attr.PI_ABORTED]).toBe(true); expect(llm.attributes[Attr.ERROR_TYPE]).toBe("aborted"); }); @@ -359,7 +359,7 @@ describe("Pi versions without agent_settled (before 0.80.5)", () => { session.emit(cancelledRetry()); // session.abort() / abortRetry() during the retry sleep // Ended before the next prompt and without dispose: complete, ERROR with the failed call's status. const failed = one(exporter.getFinishedSpans(), SpanNames.AGENT_RUN); - expect(failed.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 503 overloaded_error" }); + expect(failed.status).toEqual({ code: SpanStatusCode.ERROR }); expect(failed.attributes[Attr.ERROR_TYPE]).toBe("overloaded_error"); expect(failed.attributes[Attr.PI_INCOMPLETE]).toBeUndefined(); expect(failed.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(1); @@ -389,7 +389,7 @@ describe("Pi versions without agent_settled (before 0.80.5)", () => { const agent = one(spans, SpanNames.AGENT_RUN); expect(agent.attributes[Attr.PI_RUN_ATTEMPTS]).toBe(2); expect(agent.attributes[Attr.PI_RUN_RETRY_COUNT]).toBe(1); - expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "error: HTTP 503 overloaded_error" }); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR }); expect(spans).toHaveLength(5); // run, 2 turns, 2 model calls expect(new Set(spans.map((s) => s.spanContext().traceId)).size).toBe(1); }); @@ -422,7 +422,7 @@ describe("Pi versions without agent_settled (before 0.80.5)", () => { expect(byName(exporter.getFinishedSpans(), SpanNames.AGENT_RUN)).toHaveLength(0); session.emit([{ type: "agent_settled", aborted: true }]); const agent = one(exporter.getFinishedSpans(), SpanNames.AGENT_RUN); - expect(agent.status).toEqual({ code: SpanStatusCode.ERROR, message: "aborted" }); + expect(agent.status).toEqual({ code: SpanStatusCode.ERROR }); expect(agent.events.map((e) => e.name)).toEqual(["pi.auto_retry.start", "pi.auto_retry.end"]); }); }); @@ -467,7 +467,7 @@ describe("compaction", () => { it("marks a failed compaction ERROR without its error text by default", () => { const failure = `summarization failed reading ${LOCAL_PATH}`; const hidden = one(run(manualCompaction(failure)).spans, SpanNames.COMPACTION); - expect(hidden.status).toEqual({ code: SpanStatusCode.ERROR, message: "compaction failed" }); + expect(hidden.status).toEqual({ code: SpanStatusCode.ERROR }); expect(exportedText([hidden])).not.toContain(LOCAL_USER); const shown = one(run(manualCompaction(failure), CONTENT_ON).spans, SpanNames.COMPACTION); expect(shown.status.message).toBe(failure); @@ -531,7 +531,7 @@ describe("content policy", () => { expect(inputsOnly).not.toContain(RESULT_SECRET); expect(inputsOnly).not.toContain(OUTPUT_SECRET); - const outputsOnly = exportedText(run(happyPath(), { traceConfig: { hideOutputs: false } }).spans); + const outputsOnly = exportedText(run(happyPath(), { traceConfig: { hideOutputs: false, hideErrorDetails: false } }).spans); expect(outputsOnly).not.toContain(PROMPT_SECRET); expect(outputsOnly).not.toContain(ARG_SECRET); expect(outputsOnly).toContain(RESULT_SECRET); diff --git a/typescript/packages/traceai_pi_coding_agent/src/content.ts b/typescript/packages/traceai_pi_coding_agent/src/content.ts index 772bd55cf..b820d1303 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/content.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/content.ts @@ -9,6 +9,8 @@ export interface ContentPolicy { hideInputs: boolean; /** Drop assistant text and tool results. Tool-call arguments follow `hideInputs`. */ hideOutputs: boolean; + /** Drop error text independently of ordinary output capture. */ + hideErrorDetails: boolean; /** Longest exported text value, in characters. Longer values are cut and marked. */ maxContentLength: number; } diff --git a/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts b/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts index 190ecc27d..6cadcd531 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/instrumentation.ts @@ -22,6 +22,7 @@ export const INSTRUMENTATION_NAME = "@traceai/pi-coding-agent"; */ const FI_HIDE_INPUTS = "FI_HIDE_INPUTS"; const FI_HIDE_OUTPUTS = "FI_HIDE_OUTPUTS"; +const FI_HIDE_ERROR_DETAILS = "FI_HIDE_ERROR_DETAILS"; export interface PiCodingAgentInstrumentationConfig { /** Provider to create spans with. Defaults to the global provider at call time. */ @@ -54,6 +55,7 @@ export function resolveContentPolicy( return { hideInputs: traceConfig?.hideInputs ?? envHide(FI_HIDE_INPUTS), hideOutputs: traceConfig?.hideOutputs ?? envHide(FI_HIDE_OUTPUTS), + hideErrorDetails: traceConfig?.hideErrorDetails ?? envHide(FI_HIDE_ERROR_DETAILS), maxContentLength, }; } @@ -134,7 +136,7 @@ export class PiCodingAgentInstrumentation { const provider = this.tracerProvider ?? trace.getTracerProvider(); const tracer = new FITracer({ tracer: provider.getTracer(INSTRUMENTATION_NAME, VERSION), - traceConfig: { ...this.traceConfig, hideInputs: policy.hideInputs, hideOutputs: policy.hideOutputs }, + traceConfig: { ...this.traceConfig, hideInputs: policy.hideInputs, hideOutputs: policy.hideOutputs, hideErrorDetails: policy.hideErrorDetails }, }); const sessionTracer = new PiSessionTracer(session, tracer, policy); sessionTracer.attach(); diff --git a/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts index bf8d9fa3d..da969ee1a 100644 --- a/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts +++ b/typescript/packages/traceai_pi_coding_agent/src/sessionTracer.ts @@ -297,7 +297,7 @@ export class PiSessionTracer { private setError(span: Span, details: { type: string; message: string }): void { span.setAttribute(Attr.ERROR_TYPE, details.type); - span.setStatus({ code: SpanStatusCode.ERROR, message: details.message }); + span.setStatus({ code: SpanStatusCode.ERROR, ...(this.policy.hideErrorDetails ? {} : { message: details.message }) }); } private ensureRun(): RunState { @@ -349,7 +349,8 @@ export class PiSessionTracer { span.setAttribute(Attr.INPUT_MIME_TYPE, MimeType.TEXT); } const last = run.lastAssistant; - if (last && !this.policy.hideOutputs) { + if (last && !this.policy.hideOutputs && + (!this.policy.hideErrorDetails || !["error", "aborted"].includes(last.stopReason ?? ""))) { span.setAttribute(Attr.OUTPUT_VALUE, truncate(messageText(last.content), this.policy.maxContentLength)); span.setAttribute(Attr.OUTPUT_MIME_TYPE, MimeType.TEXT); } @@ -371,7 +372,7 @@ export class PiSessionTracer { if (aborted) { this.setError(span, { type: "aborted", message: "aborted" }); } else if (failed) { - this.setError(span, errorDetails(last, !this.policy.hideOutputs)); + this.setError(span, errorDetails(last, !this.policy.hideOutputs && !this.policy.hideErrorDetails)); } else if (!incomplete) { span.setStatus({ code: SpanStatusCode.OK }); } @@ -404,7 +405,7 @@ export class PiSessionTracer { [Attr.PI_INCOMPLETE]: incomplete || undefined, }); if (message?.stopReason === "error" || message?.stopReason === "aborted") { - this.setError(turn.span, errorDetails(message, !this.policy.hideOutputs)); + this.setError(turn.span, errorDetails(message, !this.policy.hideOutputs && !this.policy.hideErrorDetails)); } else if (!incomplete) { turn.span.setStatus({ code: SpanStatusCode.OK }); } @@ -477,7 +478,8 @@ export class PiSessionTracer { if (!incomplete) { const output = num(usage.output); setUsage(span, usage); - if (!this.policy.hideOutputs) { + if (!this.policy.hideOutputs && + (!this.policy.hideErrorDetails || !["error", "aborted"].includes(final.stopReason ?? ""))) { const text = messageText(final.content); // Tool-call arguments are the tool's input, so they follow hideInputs. const value = calls.length @@ -504,7 +506,7 @@ export class PiSessionTracer { } } if (final.stopReason === "error" || final.stopReason === "aborted") { - this.setError(span, errorDetails(final, !this.policy.hideOutputs)); + this.setError(span, errorDetails(final, !this.policy.hideOutputs && !this.policy.hideErrorDetails)); if (final.stopReason === "aborted") span.setAttribute(Attr.PI_ABORTED, true); } else if (!incomplete) { span.setStatus({ code: SpanStatusCode.OK }); @@ -558,7 +560,8 @@ export class PiSessionTracer { [Attr.PI_TOOL_DURATION_MS]: num(event.durationMs), [Attr.PI_TOOL_IS_ERROR]: isError, }); - const resultText = this.policy.hideOutputs ? undefined : toolResultText(event.result); + const resultText = this.policy.hideOutputs || (isError && this.policy.hideErrorDetails) + ? undefined : toolResultText(event.result); if (resultText !== undefined) { span.setAttribute(Attr.OUTPUT_VALUE, truncate(resultText, this.policy.maxContentLength)); span.setAttribute(Attr.OUTPUT_MIME_TYPE, MimeType.TEXT); @@ -620,7 +623,8 @@ export class PiSessionTracer { run.cost += num(usage.cost?.total) ?? 0; } } - if (!this.policy.hideOutputs && str(result?.summary)) { + if (!this.policy.hideOutputs && str(result?.summary) && + (!this.policy.hideErrorDetails || (!event.errorMessage && event.aborted !== true))) { span.setAttribute(Attr.OUTPUT_VALUE, truncate(result!.summary as string, this.policy.maxContentLength)); span.setAttribute(Attr.OUTPUT_MIME_TYPE, MimeType.TEXT); } @@ -630,7 +634,7 @@ export class PiSessionTracer { } else if (errorMessage) { this.setError(span, { type: "compaction_error", - message: this.policy.hideOutputs ? "compaction failed" : truncate(errorMessage, ERROR_TEXT_LIMIT), + message: this.policy.hideOutputs || this.policy.hideErrorDetails ? "compaction failed" : truncate(errorMessage, ERROR_TEXT_LIMIT), }); } else { span.setStatus({ code: SpanStatusCode.OK }); From a4571e43098a27f77ceffbc7c246df5e12c3af78 Mon Sep 17 00:00:00 2001 From: Nikhil Pareek <8347993+nik13@users.noreply.github.com> Date: Mon, 26 Oct 2026 11:49:06 +0000 Subject: [PATCH 6/7] ci: use the exact JDK patch release supported by setup-java Temurin labels the build metadata with an LTS suffix; pin the unambiguous 21.0.8 release instead of an unavailable 21.0.8+9 variant. Co-Authored-By: Claude Code --- .github/workflows/integration-tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index 7546c0451..c767da2be 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -182,7 +182,7 @@ jobs: - uses: actions/setup-java@c5195efecf7bdfc987ee8bae7a71cb8b11521c00 # v4.7.1 with: distribution: temurin - java-version: '21.0.8+9' + java-version: '21.0.8' - name: Install checksum-pinned Maven 3.9.9 run: bash .github/integration-ci/install_maven.sh - name: Build packaged application and run OTLP integration tests From 7f0944b0cc510a1e36919965df4e3f32dafcf258 Mon Sep 17 00:00:00 2001 From: nik13 Date: Mon, 26 Oct 2026 21:49:51 -0700 Subject: [PATCH 7/7] ci: enable the pinned pnpm shim for nested package scripts Clean-runner validation exercises installation without a global pnpm, then builds dependencies and the target, runs the real unit/contract tests and packs the package with the shim on PATH. Co-Authored-By: Claude Code --- .github/workflows/integration-tests.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml index c767da2be..de45da74b 100644 --- a/.github/workflows/integration-tests.yml +++ b/.github/workflows/integration-tests.yml @@ -109,6 +109,11 @@ jobs: env: PACKAGE_NAME: ${{ matrix.package.name }} run: | + # package scripts invoke `pnpm` themselves; setup-node supplies + # corepack, but does not enable its pnpm shim on a clean runner. + mkdir -p "$RUNNER_TEMP/corepack-bin" + corepack enable --install-directory "$RUNNER_TEMP/corepack-bin" pnpm + echo "$RUNNER_TEMP/corepack-bin" >> "$GITHUB_PATH" # packageManager pins pnpm 10.30.1 with its integrity hash. corepack pnpm --version corepack pnpm --filter "@traceai/$PACKAGE_NAME..." install --frozen-lockfile --ignore-scripts