Repository navigation
feat(pi-coding-agent): Pi Coding Agent SDK instrumentor (TH-8264) - #252
Merged
Merged
Conversation
@traceai/pi-coding-agent traces @earendil-works/pi-coding-agent sessions through AgentSession.subscribe(): one AGENT span per agent run (agent_start..agent_settled, across Pi's automatic retries), a CHAIN span per turn, an LLM span per assistant message (model, provider, tokens with cache reads added back, cost when priced, finish reason, ERROR on error/aborted) and a TOOL span per tool execution (nested by parentToolCallId, ERROR on isError). session.id is the Pi session id on every span. The SDK is ESM-only, so apps wrap createAgentSession or call instrumentSession(session). Prompts, tool arguments, tool results and answers are hidden by default (FI_HIDE_INPUTS/FI_HIDE_OUTPUTS=false or traceConfig opt in); the system prompt is never exported. No runtime or lockfile dependency on the SDK: sessions are read structurally, and contract/sdk_types.ts checks the real 1.2.0 types fit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 15, 2026
… error privacy (TH-8264) - Run end on Pi before 0.80.5 (no agent_settled): end the run at an agent_end whose willRetry is not true, until the session is seen to settle. Sessions with AgentSession.waitForIdle() (added with agent_settled in 0.80.5) end at agent_settled from the start. Two prompts on old Pi now make two complete traces without dispose(). - Compaction is a pi.compaction CHAIN span with reason, tokens before/after and the summarization call's gen_ai.usage.* and cost; inside a run it is a child (and counts in run totals), session.compact() / pre-prompt compaction is its own one-span trace with the same session.id. - Error privacy: with outputs hidden, error statuses carry only the stop reason, HTTP status and provider error type (error.type set); the raw provider, tool and compaction error text needs hideOutputs: false. Only vendor-style error types are taken from the body, never an echoed "type". - pi.session.name (user-authored) only with hideInputs: false. - finish_reasons maps toolUse to tool_calls and drops pending; children end before parents on dispose; tool end without toolCallId closes its own span. - stop() removes the per-session entry and restores dispose(), so a session can be traced again; Pi's dispose() runs even if ending spans throws. - pi.agent.attempts defined as agent loops; new pi.agent.retry_count. - README: supported Pi versions, always-exported fields, double traces with the Pi extension, register() throw; example/contract use read+ls only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Pi 0.75.4 to 0.80.3 (no agent_settled), session.abort() or abortRetry() during the retry backoff emits auto_retry_end with success:false after an agent_end that announced the retry, and no further agent_start. The run stayed open, so the next prompt joined the failed run's trace and the run ended OK. End the run at that auto_retry_end when the session does not settle. Sessions that settle (Pi 0.80.5+) still end at agent_settled. Tests: a legacy cancelled retry gives a complete ERROR run before the next prompt and a separate trace for that prompt; exhausted retries still end at the last agent_end; a settling session waits for agent_settled. Checked against real Pi 0.75.4 and 0.80.3 via both abort() and abortRetry(). Docs: README error-status wording now names the error class-name fallback and the "status code NNN" form; the contract README uses the self-hosted collector port 4318; test fixtures use a neutral model and provider name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13
marked this pull request as ready for review
October 21, 2026 22:19
Co-Authored-By: Claude Code <noreply@anthropic.com>
Plan only supported packages present at the head. Preserve Python and LiveKit coverage; use frozen dependencies, pinned toolchains and a fail-closed aggregate. No vendor, publication or deployment credentials. Co-Authored-By: Claude Code <noreply@anthropic.com>
Keep default error descriptions private even when outputs are enabled. Preserve statuses/types, require both opt-ins for raw errors, and cover custom attributes, failed tool results, retries and exception paths with regression tests. Co-Authored-By: Claude Code <noreply@anthropic.com>
Temurin labels the build metadata with an LTS suffix; pin the unambiguous 21.0.8 release instead of an unavailable 21.0.8+9 variant. Co-Authored-By: Claude Code <noreply@anthropic.com>
Clean-runner validation exercises installation without a global pnpm, then builds dependencies and the target, runs the real unit/contract tests and packs the package with the shim on PATH. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@traceai/pi-coding-agent(new package,typescript/packages/traceai_pi_coding_agent) traces Pi Coding Agent SDK sessions (createAgentSession()/AgentSession) into Future AGI. TH-8264.AgentSession.subscribe()events. Tree per agent run (session.prompt(), including Pi's automatic retries, continuations and in-run compaction):invoke_agent pi(AGENT,agent_start..agent_settled) >pi.turn(CHAIN) >chat <model>(LLM, per assistant message) andexecute_tool <tool>(TOOL; nested under the calling tool when the event hasparentToolCallId), pluspi.compaction(CHAIN) when Pi compacts during the run.agent_settled; 0.80.4 is not on npm) end a run atagent_settled. 0.75.4 to 0.80.3 have noagent_settled: the run ends at the firstagent_endwhosewillRetryis nottrue, or at anauto_retry_endwithsuccess: falsewhen a retry is cancelled during its backoff (retries stay in the run; a continuation after that point is a second trace). Before 0.75.4 (all@mariozechner/*)agent_endhas nowillRetry, so each retry is its own trace. A session is treated as settling from the start when it hasAgentSession.waitForIdle(), which Pi added in the same release asagent_settled(checked in the 0.75.3, 0.80.3, 0.80.5 and 1.2.0 tarballs).pi.compaction: reason, tokens before / estimated after, and the summarization call'sgen_ai.usage.*and cost (counted in run totals).session.compact()and the compaction Pi runs before a prompt are their own one-span trace with the samesession.id.session.id= PisessionIdon every span, so runs of one session group in Sessions.pi.api, tokens (input = Piinput+cacheRead+cacheWrite; Pi's own uncachedinputkept aspi.usage.uncached_input_tokens), cache read/write, reasoning,gen_ai.cost.totalonly when Pi priced the call,gen_ai.response.finish_reasons(toolUsereported as OTeltool_calls; nothing whilepending),pi.stop_reason, tool-call names. Start/end from the message's owntimestamp+durationMs.stopReasonerror/aborted-> LLM + turn ERROR witherror.type; run ERROR when aborted or its last model call failed (a failed call Pi retried successfully leaves the run OK); toolisError-> ERROR; failed compaction -> ERROR. With content off the status message is built only from the stop reason, the HTTP status and the provider's error type or, without one, the error's class name (for exampleerror: HTTP 400 invalid_request_error,error: TypeError); the raw provider / tool / compaction error text needshideOutputs: false.dispose()mid-run ends open spans, children first, withpi.incomplete=true. Listeners never throw into Pi, and Pi'sdispose()runs even if ending spans throws.wrapCreateAgentSession(createAgentSession)orinstrumentSession(session)(no module patching). One instrumentation per session; the stop function removes it so the session can be traced again. No runtime or lockfile dependency on the SDK: sessions are read structurally;contract/sdk_types.tsproves the real 1.2.0AgentSession/createAgentSessiontypes fit (re-checked after these changes).@traceai/claude-agent-sdk(feat(claude-agent-sdk): @traceai/claude-agent-sdk for TypeScript (TH-8235) #209), stricter on error text: prompts, tool arguments, tool results, answers, the session name, the compaction summary and all raw error text are hidden by default;traceConfig: { hideInputs: false, hideOutputs: false }orFI_HIDE_INPUTS=false/FI_HIDE_OUTPUTS=false(fail-closed: onlyfalse, any case, trimmed) opts in. The system prompt is never exported.pi.agent.*keys, notgen_ai.usage.*, so session/dashboard token sums do not double count.pi.agent.attempts= agent loops in the run (first + retries + continuations);pi.agent.retry_count= automatic retries.pnpm install --frozen-lockfilepasses. README catalog rows untouched.Review fixes (2026-10-15, commit bc61065)
agent_settled, so runs never closed and prompts mergedagent_endwithoutwillRetry; README "Supported Pi versions" row; tests for two legacy prompts (two complete traces without dispose), a legacy retry, and the switch toagent_settledpi.compactionspan with usage, in-run and standalone; testspi.session.namebehindhideInputs; error statuses built from stop reason + HTTP status + provider error type; raw text behindhideOutputs; tests with a path + username in provider and tool errors assert absence by defaulttoolCallIdleft duplicate UNSET spans["pending"]finish reasons;toolUseinstead oftool_calls; turn ended before its childrentool_calls, end children before parentsinstrumentSession()afterstop()did nothingdispose; documented one-instrumentation-per-sessiondisposewrapper could skip Pi'sdispose()detach()never throwspi.agent.attemptsundefinedpi.agent.retry_countReview fixes, round 2 (2026-10-15, commit 8067247)
session.abort()/abortRetry()during the backoff) left the run open, so the next prompt merged into the failed run's trace and the run ended OKauto_retry_endwithsuccess: falseends the run when the session does not settle; the run is ERROR with the failed call's status. Sessions withagent_settled(0.80.5+) still end there. Exhausted retries already ended at the lastagent_end, so that path is unchanged. Tests: legacy cancelled retry (complete ERROR run before the next prompt, next prompt its own trace), legacy exhausted retries (one run), settling session waits foragent_settled. The cancelled-retry test fails without the fix. Real Pi 0.75.4 and 0.80.3 with a mock gateway returning 503, via bothabort()andabortRetry(): after prompt 1 the run has ended (error: HTTP 503 ..., attempts 1, retry_count 1), and prompt 2 is a second trace with an OK run. Recovered retries, exhausted retries and two plain prompts give the same results as beforestatus code NNNform and the error class-name fallbackerror.typerow and privacy line now say both;errorDetails()doc comment matchescontract/README.mdused a dev-machineFI_BASE_URLporthttp://localhost:4318, marked self-hosted onlyexample-model/my-gatewayHow to run
Real-SDK checks:
typescript/packages/traceai_pi_coding_agent/contract/README.md.E2E evidence (local Future AGI, real Pi SDK, real model) — re-run 2026-10-15 at bc61065
@earendil-works/pi-coding-agent1.2.0, Node 26.8.1, published@traceai/fi-core1.0.0 from npm, this package packed withpnpm packat bc61065. Modelgpt-6-lunathrough an OpenAI-compatible gateway (Pimodels.json,api: openai-completions), isolatedPI_CODING_AGENT_DIR, throwaway project with a README.md. Run with the docs exampleexamples/coding-agents/pi-sdk/run_e2e.sh(docs(coding-agents): Pi Coding Agent SDK tracing (TH-8264) docs#916), with a relative workdir and a relative tarball path.lsthenread, answered "Throwaway project".th8103-coding-pi-sdk, trace07612edc-fda7-40b3-9802-fe451366a1bb:42b8f724-ec4a-47ea-a626-037d8456567e(unknown model id; the gateway returned HTTP 400): run, turn andchat no-such-model-th8103are allERROR, status messageerror: HTTP 400 invalid_request_error,error.type=invalid_request_error; the gateway's text is not stored.run_e2e.shexited 1. PASS.FI_HIDE_INPUTS=FALSE FI_HIDE_OUTPUTS=" False ", tracec75d0f89-ff06-4063-b5df-0a6b25af30ca: status message is the gateway's raw400: {"message":"unknown provider for model ...","type":"invalid_request_error",...}and the prompt marker is on the run span (mixed-case, padded env values opt in).session.compact(), real summarization call): prompt trace8c8a0119-97b8-4cbd-8ca8-4d8ec60ab6e4(run > 2 turns > chat + read), and compaction trace895b3291-7589-45c3-9820-5e8ae93f1b03: one rootpi.compactionCHAIN span,pi.compaction.reason=manual, samesession.id, tokens 545/122/667. PASS.Privacy proof
/Users/0,/tmp/0, README text 0; every spanin=0B out=0B.1/yes/truedo not opt in.Limitations / not available
models.jsonmodels without prices report 0 and get no cost key.AgentSessionRuntimereplacement (new/resume/fork/import) creates a newAgentSession; the app must callinstrumentSession()again (Pi's docs say the same for any subscription).bindExtensions(), every prompt is traced twice; README troubleshooting shows aDefaultResourceLoaderfilter.@traceai/pi-coding-agentis not on npm yet; docs#916 says so on the page and shows how to install from this PR.🤖 Generated with Claude Code