diff --git a/README.md b/README.md index 7ee65f632..f20794abe 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Trace LLM calls, agents, tools, and retrieval. Keep your application code and yo **79 integration families · 5 languages · OpenTelemetry-native · Apache 2.0** -[Get started](#quickstart) · [Integrations](#integrations) · [Documentation](https://docs.futureagi.com/docs/tracing) · [Contribute](CONTRIBUTING.md) +[Zero-code Node.js](#zero-code-tracing-for-nodejs) · [SDK quickstarts](#quickstart) · [Integrations](#integrations) · [Documentation](https://docs.futureagi.com/docs/tracing) · [Contribute](CONTRIBUTING.md) [![PyPI](https://img.shields.io/pypi/v/fi-instrumentation-otel?label=PyPI&color=555555)](https://pypi.org/project/fi-instrumentation-otel/) [![npm](https://img.shields.io/npm/v/%40traceai%2Ffi-core?label=npm&color=555555)](https://www.npmjs.com/package/@traceai/fi-core) @@ -31,9 +31,24 @@ Trace LLM calls, agents, tools, and retrieval. Keep your application code and yo - **Follow the request.** Connect model calls, retrieval, and tool execution with OpenTelemetry spans and context. - **Choose what leaves your app.** Configure content masking and export through your own OpenTelemetry pipeline. +## Zero-code tracing for Node.js + +**See your AI calls without adding tracing code.** Install [`@traceai/auto`](typescript/packages/traceai_auto/) and preload it before your app (Node.js 20.6+, ESM or CommonJS): + +```bash +npm install @traceai/auto +FI_API_KEY="your-api-key" FI_SECRET_KEY="your-secret-key" \ +FI_PROJECT_NAME="my-ai-app" \ +node --import @traceai/auto/register app.mjs +``` + +Keep your provider credentials configured as usual. Make an AI request, consume the response, then open **Observe → my-ai-app** in [Future AGI](https://app.futureagi.com). OpenAI instrumentation is included; recognized AI calls through global `fetch` are captured too. Optional framework integrations retain their own coverage limits. + +**Prompts and responses are captured by default.** Set `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true FI_HIDE_ERROR_DETAILS=true` before startup to hide content. [Setup, supported libraries, privacy and framework recipes →](typescript/packages/traceai_auto/) + ## Quickstart -Pick your language. These examples send traces to [Future AGI](https://app.futureagi.com); you can also supply your own OpenTelemetry tracer provider and exporter. +Prefer explicit instrumentation or another language? Pick your language. These examples send traces to [Future AGI](https://app.futureagi.com); you can also supply your own OpenTelemetry tracer provider and exporter. | Language | Instrumentation | Install / coordinates | | :--- | :--- | :--- | diff --git a/spec/auto-instrumentation/README.md b/spec/auto-instrumentation/README.md index a4e76eb22..1da863d98 100644 --- a/spec/auto-instrumentation/README.md +++ b/spec/auto-instrumentation/README.md @@ -33,7 +33,7 @@ It runs on the HTTP clients each runtime uses for AI calls: | SDK | Transports patched | | --- | --- | | Python | `httpx` and `httpx2` (sync and async), `requests`, `aiohttp` | -| TypeScript | `globalThis.fetch` (covers undici and every fetch-based SDK) | +| TypeScript | `globalThis.fetch` as observed after preload; not direct `undici.fetch`/`undici.request`, `node-fetch`, or a fetch reference cached before preload | | Go | `http.DefaultTransport` (and any `http.Client` built from it) | | C# | `HttpClient` through a `DiagnosticListener` handler | | Java | `java.net.http.HttpClient` and OkHttp when present | diff --git a/typescript/packages/fi-core/README.md b/typescript/packages/fi-core/README.md index 4d54771d3..974c1705b 100644 --- a/typescript/packages/fi-core/README.md +++ b/typescript/packages/fi-core/README.md @@ -2,6 +2,12 @@ Core OpenTelemetry instrumentation library for TypeScript applications with advanced evaluation capabilities for AI systems. +## Choose your setup + +For **zero-code Node.js tracing**, start with [`@traceai/auto`](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/README.md): install one package and preload it before your app. It includes OpenAI instrumentation, recognizes supported AI traffic through global `fetch`, and enables eligible installed integrations. Its own export route is batched; prompts and responses are captured by default, with explicit hide controls. + +Use **`@traceai/fi-core`** when you want to configure the provider, create custom spans or register instrumentors yourself. Do not add a second provider or duplicate instrumentation to an app already using the auto preload. Application-owned OpenTelemetry exporters keep their own privacy policies. + ## Overview `@traceai/fi-core` provides a comprehensive tracing solution built on OpenTelemetry that's specifically designed for AI applications. It offers custom span exporters, evaluation tags, and seamless integration with the TraceAI platform for observability and performance monitoring. diff --git a/typescript/packages/traceai_auto/README.md b/typescript/packages/traceai_auto/README.md index cc59456e7..d3860dd39 100644 --- a/typescript/packages/traceai_auto/README.md +++ b/typescript/packages/traceai_auto/README.md @@ -1,736 +1,253 @@ # @traceai/auto -Zero-code AI tracing for Node.js 20.6+ (ESM and CommonJS). Two complementary -layers cover installed traceAI integrations and recognized AI HTTP wire protocols. -This is broad process-level coverage, **not a claim that every possible AI client, -transport, or framework is supported**. See the limitations below. +**Trace your Node.js AI calls without changing your application code.** Preload once to capture supported model requests, responses, token usage and timing as OpenTelemetry spans. -## Install and run +[Quick start](#quick-start) · [Coverage](#supported-libraries-and-protocols) · [Privacy](#privacy-defaults) · [Frameworks](#frameworks-and-deployment) · [Troubleshooting](#troubleshooting) + +## Quick start + +You need **Node.js 20.6+**, an existing Node application, and a [Future AGI](https://app.futureagi.com) API key / secret key pair. Keep your model provider's credentials configured as usual; traceAI does not supply a model key. + +Install in your application's directory: ```sh npm install @traceai/auto -export FI_API_KEY=your-api-key -export FI_SECRET_KEY=your-secret-key -node --import @traceai/auto/register app.mjs ``` -Other entry points use the same singleton: +Set both FI keys through your deployment's secret manager or shell environment, then start the app with one preload: ```sh -NODE_OPTIONS="--import @traceai/auto/register" node app.js -node -r @traceai/auto/register app.cjs +FI_API_KEY="your-api-key" FI_SECRET_KEY="your-secret-key" \ +FI_PROJECT_NAME="my-ai-app" \ +node --import @traceai/auto/register app.mjs ``` -Or initialize before importing AI libraries: +Replace `app.mjs` with your entry file. **Prompts and responses are captured by default.** To hide them, set `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true FI_HIDE_ERROR_DETAILS=true` before startup. See [privacy defaults](#privacy-defaults) before deploying with sensitive data. -```ts -import { init } from "@traceai/auto"; -init(); -``` +Make an AI request, consume the response, then open **Observe → my-ai-app** in Future AGI. For supported calls, the trace includes the provider/model, input and output, duration, and any usage the provider returned. Streaming timing is available when the transport exposes it; missing usage is not invented. + +### Keep your existing start command -`init(options)` configures the process **once**. It retains the existing provider, -configuration and hooks on every later call, logging one diagnostic line. It returns -the **same tracer provider object**, or `undefined` when initialization is disabled or -failed. Diagnostics are returned by `report()`, not `init()`. Other exports are -`forceFlush(): Promise`, `shutdown(): Promise`, and -`registerSecret(value: unknown): void` (also exported additively by `@traceai/fi-core`). -Integrations can register credentials observed outside a transport without supplying -span context or retaining per-trace credential groups. Non-string values and strings -shorter than eight characters are ignored; registration never coerces user objects. -Shutdown removes this agent's hooks/processors but never shuts down an -application-owned global OTel provider. Reinitialization after shutdown or failed -startup, changing privacy/exporter settings, and disabling an enabled integration -by calling `init()` again are unsupported: restart the process to change configuration. -For short-lived programs, finish reading responses, then `await forceFlush()` or -`await shutdown()`. - -The preload does not perform network I/O. The first real completed span initiates -a bounded batch export, or `doctor` can explicitly send a test span. FI credentials -are required only for the default FI destination. `init({ exporter })`, -`init({ spanProcessor })`, `init({ provider })`, or an existing application OTel provider -activates instrumentation without FI credentials and without creating an FI export route. -With neither a destination nor a complete FI pair, both layers are disabled with one warning. -FI project routing never replaces application `service.name` or environment resource -identity. Export/integration failures warn once per -failure category, do not log credentials/provider bodies, and leave the app running. -A one-shot `beforeExit` handler flushes completed spans. The agent adds no SIGINT -handler. Its SIGTERM listener starts a bounded flush and preserves app-owned listeners; -without one, it restores the default action and re-raises SIGTERM (preserving the signal -exit). Explicit flush/shutdown have a five-second deadline. Shutdown reserves up to -2.5 seconds for flush and then invokes exporter shutdown with the remaining 2.5 seconds, -even if flush never resolves. A failed initialization rolls back this agent's hooks, -processor and newly installed globals; app-owned globals are retained. The first -configuration remains terminal, so a later `init()` cannot retry or rewire the -process. SIGKILL/forced termination cannot guarantee delivery. - -### Relocatable launcher payload - -The build also produces `dist/agent/`. Copy the **whole directory**, including -`runtime.cjs`, `hook.mjs`, `hook-register.cjs` and linked `.LEGAL.txt` notices. -Also copy `dist/THIRD-PARTY-NOTICES` and `dist/third-party-manifest.json` alongside -the relocated payload; the linked comments alone are not the full license texts: +With the FI variables above already set, use `NODE_OPTIONS` for an npm script or process launcher: ```sh -node --require "/absolute/agent directory/preload.cjs" app.mjs -NODE_OPTIONS='--require "/absolute/agent directory/preload.cjs"' node app.cjs +NODE_OPTIONS="--import @traceai/auto/register" npm start ``` -Use `--require`, not `--import`, for launcher injection. No `@traceai/auto` install in -the application is needed. The payload embeds OpenAI instrumentation, including support -for OpenAI v4's default node-fetch transport; app-installed instrumentors take precedence. -The ES5-only preload skips Node <18 without loading/parsing the modern payload. Node -without `module.register` keeps CJS/fetch capture, not ESM library hooks. Historical -binaries were not exercised: the unsupported-version guard is VM-tested, and a missing -`module.register` is tested in a real child process. The supported package engine is -Node >=20.6. - -## Capture layers - -### Wire layer - -The runtime reads a byte-identical packaged copy of the shared -`spec/auto-instrumentation/wire-protocols.json` catalog. It patches `globalThis.fetch`, -preserving the fetch function's name, length and custom properties via a Proxy. - -Recognized formats include: - -- OpenAI-compatible chat/completions, Responses, embeddings, image generation, - moderation, transcription/translation and speech generation; compatible gateways - and providers are recognized by the catalog. Unknown hosts use the endpoint's - protocol-specific fallback (`openai_compatible`, `anthropic_compatible`, - `gemini_compatible`, or `cohere_compatible`). -- Anthropic messages, Gemini/Vertex generate-content and embeddings, Cohere chat, - embed and rerank, Bedrock converse/invoke, Ollama chat/generate/embed. -- Deepgram listen/speak, ElevenLabs TTS/STT, AssemblyAI transcript submission and - completed polling, and Cartesia bytes/SSE TTS. - -Request-body validation prevents unrelated application endpoints from being claimed. -Request streams, opaque Request objects and over-limit bodies are captured **only for -catalog providers**, without request content and with `traceai.wire.body_unavailable` -or `traceai.wire.truncated`. Unknown proxies need a validated materialized request. -The actual fetch is invoked before inspecting materialized bodies: no request cloning, -stream read-ahead or request-side wait is added. - -Wire spans are OTel CLIENT spans with the saved parent context. Start time is the -transport handoff, before URL validation or request inspection; end time is the last -observed byte/error, captured **before** copying, decoding or parsing. Parsing is -excluded from duration. Streams are observed by tapping the existing readers and -iterators, not by replacing the body. The caller receives the **original fetch -Promise and original Response**, including subclass/helper methods and native headers, -URL/type, clone, tee, BYOB support (or its original rejection), cancellation promises, -and chunk identities. Capture copies bytes into its own bounded buffers without -transferring/detaching app storage. No read, reader lock, synthetic Response, automatic -clone or eager tee is created for an unread body. Export is never awaited between -chunks; reading multiple clone/tee branches produces at most one wire span. - -Text/JSON/SSE/NDJSON capture stops at the configured byte cap, discards buffered -content, and sets `traceai.wire.truncated`. Audio/binary responses are counted, not -buffered. JSON/SSE TTS base64 audio is counted with a bounded streaming scanner, -without accumulating or decoding audio. If a custom fetch exposes compressed JSON/SSE -TTS bytes, the compressed capture is bounded by the body cap and decoded asynchronously -after consumption; decoded chunks go straight to that scanner, not a retained audio -buffer. The decoded-work cap also applies; exceeding it sets `traceai.wire.truncated`. -FormData text fields are read independently -of upload size, each capped at 64 KiB; file bytes are never read, and only their -size/MIME/filename are retained. Catalog embedding formats record dimensions, not -vectors. Use `traceConfig.hideEmbeddingVectors` (or `FI_HIDE_EMBEDDING_VECTORS=true`) -to also hide vectors in generic JSON fallbacks and forwarded library/application spans. - -Native fetch normally decompresses responses. When a custom fetch exposes encoded -bytes, bounded asynchronous Node decoders handle gzip/deflate/Brotli before parsing, -plus zstd when Node exposes it. The Node-free layer uses Web `DecompressionStream` -for the codecs its runtime supports. Unavailable decoders -keep request-side metadata and set `traceai.wire.undecoded_encoding`. Already-decoded -JSON/SSE is not decoded twice. A transport failure mid-stream is ERROR with parsed -partial output; consumer cancellation is OK with `traceai.wire.incomplete = true`. - -Wire **span creation** is skipped for OTel-suppressed contexts, nested captures, and -LLM/EMBEDDING/RERANKER/GUARDRAIL ancestors, even through intermediate native SDK -spans. Outgoing credential observation is never skipped by these rules: fetch reads -headers/URL without inspecting suppressed request bodies. Node HTTP and Undici -diagnostic-channel subscribers also observe outgoing headers/URLs (including direct, -cached clients and non-AI routes); they do not create additional wire spans. The FI processor keeps weak live ancestry, so a model kind set after span -creation is also respected. Both `gen_ai.span.kind` and older `fi.span.kind` are -understood, including FI-wrapped spans. Uncovered CHAIN/AGENT/TOOL spans remain -eligible parents. Session, user and other fi-core context attributes are propagated. - -Wire attributes pass through fi-core's TraceConfig. Additional conservative handling -covers flattened message keys, audio transcripts, and raw body copies when content -hiding is enabled. `FI_HIDE_INPUTS`, `FI_HIDE_OUTPUTS`, `FI_HIDE_INPUT_MESSAGES`, -`FI_HIDE_OUTPUT_MESSAGES`, `FI_HIDE_INPUT_TEXT`, `FI_HIDE_OUTPUT_TEXT`, -`FI_HIDE_INPUT_IMAGES`, `FI_HIDE_ERROR_DETAILS`, and `FI_PII_REDACTION` are supported. -TTS `gen_ai.audio.transcript` obeys input-hide flags; STT obeys output-hide flags. -Hidden transcripts use `[REDACTED]`, exactly like hidden input/output values. - -### Privacy - -One export-time boundary sanitizes **every span sent through the agent's exporter**: -wire spans, dedicated-library spans and forwarded application spans. It builds a -detached snapshot of the ended native span; other exporters keep the original -span/resource/context unchanged by that export policy. In auto destination mode, -foreign attributes are never rewritten by FI policy. FI-authored attributes still -receive write-time hide flags, image caps, PII and known-secret protection, so -other exporters never receive authored content that those protections hid. -Application active context uses the native span; instrumentor writes use a separate -FI wrapper. Explicit standalone `FITracer` retains its active-wrapper behavior. -This is an **interim export-built view**, not complete capture: data the host SDK -already dropped or truncated may be omitted or marked unknown, never reconstructed. -Attributes (arrays element-wise), status descriptions, event names/attributes, -link attributes/contexts, resources, instrumentation scope name/version/attributes -and tracestate values all pass through this boundary. Nonfinite numbers become -`"NaN"`, `"Infinity"` or `"-Infinity"`. Unsupported attribute objects become type -names; user attribute getters, `toString` and `toJSON` are never called. The standard -ReadableSpan context method is supported for library-generated snapshots. - -Work is bounded across each snapshot (20,000 values, 8 MiB text, 32 array levels; -credential discovery also has depth/count limits). On a sanitizer error, content, -status descriptions, events, links, scope and resource content are dropped for that -span while its identity and timing survive. Upstream wire scrubbing is best-effort, -not the security boundary. Before TraceConfig masking, every recorded string is -scrubbed: - -- Data URLs and long base64 media fields (`data`, `bytes`, `b64_json`, `file_data`, - `audio_base64`, `images` elements) become ``. Image-generation - `b64_json` uses the format contract's `` placeholder. Raw audio is - never retained, even for tiny inline audio clips: scalar/array `audio` and - `input_audio` JSON leaves, indexed flattened fields and `inlineData.data.0` with - a sibling audio MIME type all use the same no-minimum-size audio policy. - MIME metadata and transcripts remain subject to their own content-hide controls. -- Credential-named keys are replaced with `` recursively, including - `authorization_token` in MCP server settings. The only logprob-name exception - is the explicitly bound provider-root policy below; access-key identifiers - remain masked. Catalog credential shapes are checked independently of names. -- Signed URL parameters, including AWS/GCP signatures and Azure SAS fields, have their - values replaced with ``, including percent-encoded parameter names. -- A single `globalThis[Symbol.for('traceai.secrets')]` registry is shared by installed, - embedded, CJS and ESM package copies in the process/isolate. It retains at most 1024 - values of eight or more characters, evicting the least recently registered/observed - value. Immutable longest-first snapshots are cached until membership changes. At - initialization it is seeded with configured FI keys and credential-like environment - values (names containing KEY, TOKEN, SECRET, PASSWORD, PASSWD, CREDENTIAL or AUTH). - Transport-observed header/query values, parsed credential fields and explicit - `registerSecret` calls augment it. Every exported span, including application-owned - instrumentation spans and unrelated/later traces, uses the current registry at - export time; there is no span-start credential inheritance or SDK-state inspection. - Credential headers and export-boundary credential fields use one shared list, - including `x-goog-api-key`, `anthropic-api-key` and `ocp-apim-subscription-key`. - Values never appear in diagnostics, reports or error messages. Every registered - value is removed from every exported string, including names, status, events, - links, resource/scope attributes and tracestate; other exporters remain unchanged. - Materialized bodies discarded by the capture cap, URLSearchParams values and - uncropped/duplicate FormData text fields still undergo bounded credential discovery - before disposal. File bytes are never inspected. Credential-discovery budget - exhaustion or failure while inspecting a materialized body drops all exported - content, not span identity, timing or status code; this also omits the truncation - marker on that fail-closed snapshot. Opaque, unparsed bodies instead follow the - scoped limitation below; their observable headers/URL still augment the registry. - Configured FI keys remain protected by the export policy even after LRU churn. - -For the closed, versioned `logprob_tokens` catalog paths, **logprob tokens are checked against registered secrets and credential shapes, not key-name rules**. -This exception is available only from the parsed OpenAI chat response root at the -instrumentation handoff: `choices[*].logprobs.content[*].token` and -`choices[*].logprobs.content[*].top_logprobs[*].token`. It does not apply to nested -JSON in user content, tool output, arbitrary attributes, or flattened key names. -If value masking changes any choice's `message.content` or `delta.content`, its -**entire logprobs block** is masked, including alternatives and byte arrays. -Buffered streaming responses are scanned by choice index before export; once a -choice is masked, earlier buffered blocks and all later blocks for that choice -are masked too. Concatenated deltas are checked for split secrets, and affected -delta text is masked. A masked individual token also loses its byte-array alias. -The staged OpenAI instrumentor exports the buffered response once, not each delta. -Other provider structures have no new exception without catalog/source evidence. - -Hide policies match semantic families, including `input.*`, `output.*`, -`gen_ai.input.*`, `gen_ai.output.*`, system instructions, tools, image/file URIs and -side-specific audio transcripts. `FI_HIDE_LLM_INVOCATION_PARAMETERS=true` additionally -hides `gen_ai.request.parameters`, explicit invocation parameters and tool schemas. -`hideEmbeddingVectors` also covers JSON fallbacks such as generic Bedrock invoke, -not only spans classified as EMBEDDING. Image hiding conservatively hides raw input -and input-message families to prevent URI aliases escaping the image policy. - -Do not put arbitrary secrets in prompts: unknown free-text secrets cannot always be -recognized. These policies cover dedicated-instrumentor spans only when they pass -through **this agent's** exporter; unrelated exporters retain their own policy. - -### Batching and Lambda lifecycle - -| Entry | Processor policy | Lifecycle | -| --- | --- | --- | -| Manual fi-core `register()` / `FITracerProvider`, omitted `batch` or explicit `batch: false` | `SimpleSpanProcessor`, preserving the manual dev default | Await provider flush/shutdown before process termination | -| Manual `batch: true` | Bounded `BatchSpanProcessor` | Await provider flush/shutdown | -| Zero-code owned provider/export route | Explicit `BatchSpanProcessor` / batch request | Auto bounded exit/signal drain; pass the returned provider to the Lambda wrapper | -| Explicit fi-core `spanProcessors` / auto `spanProcessor` | Supplied processors retain their selected policy | No silent replacement or rebatching; retain the documented owner lifecycle | - -For Lambda, create/reuse the provider outside the handler and pass the returned -provider to fi-core's `traceLambdaHandler(handler, { provider })`. The wrapper -ends its invocation span and flushes within the invocation budget; it does **not** -shut down a warm provider per invocation. R-C07 is superseded by **zero-code owned -provider batches; explicit `batch: false` honoured**. Batching and a successful -local handoff are not proof of collector acceptance. - -### Library layer - -`src/integrations.json` inventories all 45 instrumentation packages currently in the -TypeScript workspace, with their exported classes, target libraries and actual -module-hook entry points. OpenAI instrumentation is a dependency and is embedded in -the launcher payload; other dedicated packages are **not** installed implicitly. -Install the integrations you need, for example: +Preserve any other Node flags your deployment already needs. For CommonJS, either use the same `--import` preload or: ```sh -npm install @traceai/langchain +node --require @traceai/auto/register app.cjs ``` -Target/instrumentation resolution is from the application's working directory. -Install packages in that app and run from its directory. Hook registration is eager; -target patching occurs on import/require. The import-in-the-middle loader is registered -before `registerInstrumentations`, restricted to target entry points rather than -all ESM internals (which can break SDK shim live bindings). Node 26 currently emits -a deprecation warning for upstream `module.register()` when library hooks are enabled. -Both ESM and CJS patching are tested against the real OpenAI npm package, with and -without app-installed instrumentation. OpenAI v4/node-fetch can additionally emit -Node's upstream `punycode` deprecation. Neither warning is suppressed by the agent. -Disabled integrations are never loaded or registered, including integrations sharing -a target (for example DeepSeek and OpenAI). - -| Framework integration | 11-11 support | Entry point | -| --- | --- | --- | -| Mastra | Manual tracing only; no zero-code bridge at 11-11 | `createFIObservability` / `createFIMastraExporter` from `@traceai/mastra` | - -The Mastra zero-code bridge follows after 11-11. The registry reports installed -Mastra targets as `manual` and does not load `@traceai/mastra/auto` or call an enable hook. - -For Genkit and Vercel AI SDK, the registry calls an installed `enable` -entry (also accepting the integration's named enable function). It prefers the -optional `/auto` subpath. Versions without that entry are reported `manual`, not -forced into an incompatible constructor; a broken entry fails open. The entry -receives `{ tracerProvider, spanProcessor, traceConfig }` for the selected agent -branch; older no-argument entries can ignore these optional arguments and use the -initialized agent. An instrumentor returned by the entry is explicitly bound to -that provider, not an unrelated global provider. This does not replace any global -provider or synthesize FI credentials. -Manual-only integrations remain Claude Agent SDK (`wrapQuery`), Codex SDK -(`wrapCodex`/`manuallyInstrument`), Pi Coding Agent (session wrapper), and VoltAgent -(processor). Their underlying fetch AI calls can still be traced. -Existing dedicated instrumentations retain their supported-version and method -coverage constraints; registration does not expand those APIs. OpenAI Agents uses -additive processor registration (`exclusiveProcessor:false`), preserving application -processors. Shutdown recognizes processor/class-registry ownership as well as -shimmer wrappers, and leaves application-owned patches intact. - -If `@opentelemetry/instrumentation-http` is resolvable in the app, it is enabled for -incoming SERVER spans only. Outgoing non-AI HTTP calls are not traced by default: +Run from the application's directory so target libraries and optional instrumentors can be resolved. Do not add a second preload or manually patch the same client. + +### Try it with OpenAI + +If you do not have an app yet, install the OpenAI SDK (versions 4 to 7 are supported): ```sh -npm install @opentelemetry/instrumentation-http +npm install openai ``` -### Existing OTel SDKs - -An existing global SDK provider is retained, with the FI processor added beside its -existing processors. Its sampler/context manager remain authoritative. OTel JS 2.x -has no public late-add-processor API, so this uses the live MultiSpanProcessor list; -the public 1.x API is also recognized. FI project resource attributes are enriched -only on the FI export copy. If a custom provider does not expose either supported -attachment mechanism, the agent warns and uses an isolated AI provider without -replacing the global provider. This is a degraded fallback, not a promise of -processor attachment to arbitrary OTel implementations. - -## Server bundlers - -Exported adapters inject the built preload into **Node server entries only**: - -```ts -// esbuild: { platform: 'node', plugins: [traceAI()] } -import { traceAI } from '@traceai/auto/esbuild'; -// Vite: plugins: [traceAI()] — production Node SSR only, not client/webworker SSR -import { traceAI } from '@traceai/auto/vite'; -// Rollup: plugins: [traceAI({ server: true })] — explicit target required -import { traceAI } from '@traceai/auto/rollup'; -// webpack 5: { target: 'node', plugins: [new TraceAIPlugin()] } -import { TraceAIPlugin } from '@traceai/auto/webpack'; -``` +Set `OPENAI_API_KEY` and `OPENAI_MODEL` for a model your account can use. Save this ordinary application code as `app.mjs`, then use the quick-start run command. There are **no tracing imports** in the application: + +```js +import OpenAI from "openai"; -These are separate config examples, not imports to combine in one file. Adapters are -idempotent, preserve shebangs, and leave browser outputs alone. Real esbuild, Rollup, -Vite and webpack builds are tested, including plugin-only builds using external OpenAI -v4's default node-fetch transport (not just global fetch). Symlinked esbuild entries -are matched by their canonical paths; aliases, installed package entries, -extensionless entries, directory indexes and custom extensions use esbuild's own -entry-point resolver before path canonicalization. esbuild split/dynamic chunks do not get extra -preloads. CJS entries require the preload before their code while retaining strict-mode -directives (webpack uses its source-aware insertion API). A missing/broken preload warns -and leaves the application running rather than turning a deployment mistake into a crash. - -**ESM executable entries** get a tiny bootstrap at the original output path, which -initializes the agent and then dynamically imports a sibling `*.traceai-app.mjs` payload. -A static preload import alone is too late: Node has already linked the application's -static dependencies before executing it. Deploy the payload and its sourcemaps/chunks -alongside the bootstrap. The payload retains its code and mappings without an injected -line; CJS sourcemaps account for the inserted preload. - -This ESM bootstrap is for executable server entries, not library/handler modules: -exported ESM entries are rejected rather than silently losing live exports. The executing -entry's `import.meta.url` names the payload, not the bootstrap; programs relying on an -exact entry filename/main-module identity must instead use `node --require` externally. -For Next handlers, prefer `instrumentation.ts` below. Runtime/chunk code still lives in -the same directory. Bundler-specific advanced packaging modes beyond the tested cases -are unverified. - -The injected absolute path must exist at runtime: deploy `dist/agent/` to the same -path or pass `{ agentPath: '/deployed/agent/preload.cjs' }`. Do not omit sibling files. - -Preloading cannot recover package import boundaries erased by bundling. Keep AI SDKs -external for dedicated library hooks; bundled global-fetch calls still use wire capture. -Do not use these adapters for browser/edge outputs. - -### Next.js - -Prefer Next's `instrumentation.ts` with a runtime guard (works independently of the -selected Next bundler). Keep `@traceai/auto` external via `serverExternalPackages` -where your Next version supports it: - -```ts -export async function register() { - if (process.env.NEXT_RUNTIME === 'nodejs') { - await import('@traceai/auto/register'); - } -} +const response = await new OpenAI().chat.completions.create({ + model: process.env.OPENAI_MODEL, + messages: [{ role: "user", content: "Say hello in one word." }], +}); +console.log(response.choices[0].message.content); ``` -`withTraceAI(config)` from `@traceai/auto/next` is an optional **webpack-only** fallback; -it skips client/edge configs and preserves an existing webpack callback. The callback -wrapper, generated real-webpack config, and register snippet are tested. An actual -Next server build/start and Turbopack lifecycle are **unverified**; the webpack wrapper -does not cover Turbopack. Never import the Node agent in an edge instrumentation path. +The preload flushes completed spans when Node exits naturally. If your app explicitly terminates or runs inside a serverless invocation, use the [lifecycle guidance](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/FRAMEWORKS.md#aws-lambda-and-short-lived-processes). -## Edge runtime +## What you get -`@traceai/auto/edge` is an explicit fetch observer, not a zero-code Node agent: +The **Try it with OpenAI** example, as it appears in Future AGI (a real run with OpenAI SDK 7 and `openai/gpt-4.1-mini` through OpenRouter): -```ts -import { patchFetch } from '@traceai/auto/edge'; -const hook = patchFetch(4 * 1024 * 1024, (result, start, end) => { - // Adapt to the isolate's own tracer/exporter, parent context and content policy. -}); -// hook.remove() restores fetch. hook.pending contains completed-response decoding only. +![The example's chat completion span in Future AGI: model, provider, prompt and completion tokens, cost, duration, and the request and response bodies](docs/images/quickstart-trace.png) + +- **Model calls without per-call wrappers.** OpenAI instrumentation is included. Recognized AI requests through the process's global `fetch` are also captured, including compatible gateways. +- **More context from installed integrations.** Dedicated packages can add framework, agent, tool, retrieval and database spans. They retain their own method/version limits; auto-registration does not expand their coverage. +- **One trace, not two copies of each call.** Wire capture yields to covered model spans and respects OpenTelemetry suppression. Existing providers keep their sampler, context and application processors. +- **Standard OpenTelemetry.** Send to Future AGI by default, or use an [application-owned provider/exporter](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/CONFIGURATION.md#bring-your-own-opentelemetry-destination) without FI credentials. + +This is AI instrumentation, not a general-purpose APM agent. Outgoing non-AI HTTP calls, arbitrary application functions and every possible AI transport are not automatically traced. + +## Supported libraries and protocols + +**Included:** OpenAI library hooks and the global-fetch wire observer. **Optional:** other dedicated instrumentors must be installed alongside their target library. The app always owns installation of its AI SDKs. + +To see what your app needs, without installing anything: + +```sh +npx --no-install traceai-auto bootstrap ``` -Its ESM bundle has no Node APIs, Buffer, process/env access or OTel dependency. Isolated -VM tests prove JSON, rejection and streaming capture without those globals. It uses -the same catalog and privacy scrubbing. It does not install an exporter, read FI keys, -apply fi-core TraceConfig, propagate OTel context, or provide Node's suppression/dedup -layer: the adapter must supply those policies. Real Cloudflare/Vercel/Deno isolate -deployment is **unverified**. Native DecompressionStream availability is runtime-specific. +Bootstrap lists detected libraries, their status, and suggested install commands. It can also install missing instrumentors with an explicit `--install` flag; see the [CLI reference](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/CONFIGURATION.md#cli). + +### Library coverage + +The table below is generated from [`src/integrations.json`](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/src/integrations.json), the runtime registry. **Import hook** means the package registers hooks for supported target versions. **Enable entry** means automatic setup depends on the installed integration exposing its enable API. **Manual** means framework-level tracing still requires that package's explicit API; eligible underlying global-fetch calls can still be captured. + +
+All registered libraries and framework adapters + + +| Configuration name | Target library | Instrumentation package | Setup | +| --- | --- | --- | --- | +| `anthropic` | `@anthropic-ai/sdk` | `@traceai/anthropic` | Import hook | +| `bedrock` | `@aws-sdk/client-bedrock-runtime` | `@traceai/bedrock` | Import hook | +| `beeai` | `beeai-framework` | `@traceai/beeai` | Import hook | +| `cerebras` | `@cerebras/cerebras_cloud_sdk` | `@traceai/cerebras` | Import hook | +| `chromadb` | `chromadb` | `@traceai/chromadb` | Import hook | +| `claude-agent-sdk` | `@anthropic-ai/claude-agent-sdk` | `@traceai/claude-agent-sdk` | Manual | +| `codex-sdk` | `@openai/codex-sdk` | `@traceai/codex-sdk` | Manual | +| `cohere` | `cohere-ai` | `@traceai/cohere` | Import hook | +| `cursor-sdk` | `@cursor/sdk` | `@traceai/cursor-sdk` | Import hook | +| `deepseek` | `openai` | `@traceai/deepseek` | Import hook | +| `elevenlabs` | `@elevenlabs/elevenlabs-js` | `@traceai/elevenlabs` | Manual | +| `fireworks` | `openai` | `@traceai/fireworks` | Import hook | +| `genkit` | `genkit` | `@traceai/genkit` | Enable entry | +| `google-adk` | `@google/adk` | `@traceai/fi-instrumentation-google-adk` | Import hook | +| `google-genai` | `@google/generative-ai` | `@traceai/google-genai` | Import hook | +| `google-genai-sdk` | `@google/genai` | `@traceai/google-genai-sdk` | Manual | +| `groq` | `groq-sdk` | `@traceai/groq` | Import hook | +| `guardrails` | `@guardrails-ai/core` | `@traceai/guardrails` | Import hook | +| `huggingface` | `@huggingface/inference` | `@traceai/huggingface` | Import hook | +| `huggingface-transformers` | `@huggingface/transformers` | `@traceai/huggingface-transformers` | Manual | +| `instructor` | `@instructor-ai/instructor` | `@traceai/instructor` | Import hook | +| `lancedb` | `@lancedb/lancedb` | `@traceai/lancedb` | Import hook | +| `langchain` | `@langchain/core` | `@traceai/langchain` | Import hook | +| `livekit` | `livekit-client`, `@livekit/rtc-node` | `@traceai/fi-instrumentation-livekit` | Import hook | +| `llamaindex` | `llamaindex`, `@llamaindex/openai` | `@traceai/llamaindex` | Import hook | +| `mastra` | `@mastra/core`, `@mastra/observability`, `@mastra/otel-exporter` | `@traceai/mastra` | Manual | +| `mcp` | `@modelcontextprotocol/sdk` | `@traceai/mcp` | Import hook | +| `milvus` | `@zilliz/milvus2-sdk-node` | `@traceai/milvus` | Import hook | +| `mistral` | `@mistralai/mistralai` | `@traceai/mistral` | Import hook | +| `mongodb` | `mongodb` | `@traceai/mongodb` | Import hook | +| `ollama` | `ollama` | `@traceai/ollama` | Import hook | +| `openai` | `openai` | `@traceai/openai` (included) | Import hook | +| `openai-agents` | `@openai/agents` | `@traceai/openai-agents` | Import hook | +| `pgvector` | `pg`, `pgvector` | `@traceai/pgvector` | Import hook | +| `pi-coding-agent` | `@mariozechner/pi-coding-agent` | `@traceai/pi-coding-agent` | Manual | +| `pinecone` | `@pinecone-database/pinecone` | `@traceai/pinecone` | Import hook | +| `pipecat` | `@pipecat-ai/client-js` | `@traceai/fi-instrumentation-pipecat` | Import hook | +| `portkey` | `portkey-ai` | `@traceai/portkey` | Import hook | +| `qdrant` | `@qdrant/js-client-rest` | `@traceai/qdrant` | Import hook | +| `redis` | `redis` | `@traceai/redis` | Import hook | +| `strands` | `@strands-agents/sdk` | `@traceai/strands` | Import hook | +| `together` | `together-ai` | `@traceai/together` | Import hook | +| `vercel` | `ai` | `@traceai/vercel` | Enable entry | +| `vertexai` | `@google-cloud/vertexai` | `@traceai/fi-instrumentation-vertexai` | Import hook | +| `vllm` | `openai` | `@traceai/vllm` | Import hook | +| `voltagent` | `@voltagent/core` | `@traceai/voltagent` | Manual | +| `weaviate` | `weaviate-client` | `@traceai/weaviate` | Import hook | +| `xai` | `openai` | `@traceai/xai` | Import hook | + + +
+ +Dedicated instrumentors other than OpenAI are not installed implicitly. The bundled server-build adapters are inactive until you add their plugin to your build configuration. Genkit and Vercel versions without an enable entry report `manual`. Mastra is manual-only in this release. See each integration's README for supported methods and versions, and use doctor to inspect the version actually installed in your app. + +### HTTP protocol coverage + +The [wire catalog](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/src/wire/wire-protocols.json) recognizes OpenAI-compatible chat, Responses, embeddings and media endpoints; Anthropic messages; Gemini/Vertex content and embeddings; Cohere chat/embed/rerank; Bedrock converse/invoke; Ollama; and selected Deepgram, ElevenLabs, AssemblyAI and Cartesia endpoints. + +Recognition needs a matching endpoint and request shape, not just an AI-looking URL. Unknown proxy hosts can use compatible protocol fallbacks; provider-restricted endpoints still require a recognized host. Audio bytes are counted, not exported. Bedrock binary event streams are metadata-only. See [capture behavior and transport limits](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/ARCHITECTURE.md). + +## Privacy defaults + +| Surface | Default behavior | Your control | +| --- | --- | --- | +| Prompts, model outputs, tool content and transcripts | Captured where supported | `FI_HIDE_INPUTS=true`, `FI_HIDE_OUTPUTS=true`; granular message/text flags are available | +| Error details | Visible unless an input/output value, message, text or invocation-parameter hide flag implies hiding | `FI_HIDE_ERROR_DETAILS=true`; explicit `false` overrides the derived value, not secret/media masking | +| Known credentials and signed URL values | Scrubbed in traceAI's export view | Additional exact secrets can be registered with `registerSecret()`; never rely on this to detect arbitrary free-text secrets | +| PII pattern redaction | **Off** | `FI_PII_REDACTION=true`; pattern matching is not comprehensive PII detection | +| Base64 media and raw audio | Recognized payloads omitted; no audio bytes exported | Image cap defaults to `0`; increasing it does not enable base64 image export | +| Embeddings | Catalog wire formats record dimensions, not vectors | `FI_HIDE_EMBEDDING_VECTORS=true` also hides vectors on library/forwarded and generic-JSON routes | +| Extra instrumentors / server bundler plugins | Extra instrumentors are not installed; build plugins are inactive until configured | Install the integration or explicitly add a server bundler plugin | + +The agent sanitizes a detached copy for its own exporter. **Other exporters retain their own privacy policies.** Explicit application-owned processors are not silently rewritten. Host SDK limits may force the FI view to withhold content as `` rather than export a partial value. Unknown, transformed or unobserved secrets cannot always be recognized; use the hide flags for sensitive content. + +[All privacy flags and defaults](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/CONFIGURATION.md#privacy-options) · [Detailed privacy boundaries](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/ARCHITECTURE.md#privacy) ## Configuration -Precedence: code options > environment > config file > defaults. Code uses camelCase -(`projectName`, `wireCapture`, `sampleRate`, `instrumentations`, etc.); privacy flags -are under `traceConfig`. `exporter` is an optional SpanExporter override for embedding -or offline testing. It does not bypass the credential requirement. +The most-used settings are below. Set them **before** loading the agent; later `init()` calls do not reconfigure it. -| Environment | Default / meaning | +| Environment variable | Default | Purpose | +| --- | --- | --- | +| `FI_API_KEY` + `FI_SECRET_KEY` | Unset | Complete pair for the default Future AGI destination | +| `FI_PROJECT_NAME` | See full reference | Project in Observe; set explicitly for stable routing | +| `FI_BASE_URL` | `https://api.futureagi.com` | Collector base URL or full traces endpoint | +| `FI_INSTRUMENTATIONS` | All eligible integrations | Comma-separated allowlist, e.g. `openai,wire` | +| `FI_DISABLED_INSTRUMENTATIONS` | Empty | Denylist; takes precedence over the allowlist | +| `FI_WIRE_CAPTURE` | `true` | Enable global-fetch capture and transport credential observation | +| `FI_SAMPLE_RATE` | `1` | Root sampling ratio on agent-owned providers; existing providers keep their sampler | +| `FI_AUTO_DEBUG` | `false` | Print the masked startup report | +| `OTEL_SDK_DISABLED` | `false` | Disable auto-instrumentation | + +[Full configuration reference](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/CONFIGURATION.md) covers every `init()` option, environment setting, config/profile precedence, privacy flag and CLI command. + +## Frameworks and deployment + +| Environment | Start here | | --- | --- | -| `FI_API_KEY`, `FI_SECRET_KEY` | Complete pair required for the default FI destination only; prefer environment, never commit keys | -| `FI_BASE_URL` | `https://api.futureagi.com`; FI collector base or full endpoint | -| `FI_PROJECT_NAME` | `OTEL_SERVICE_NAME`, else working-directory name, else `DEFAULT_PROJECT_NAME` | -| `FI_PROJECT_TYPE` | `observe`; `experiment` is opt-in | -| `FI_INSTRUMENTATIONS` | All; comma-separated allowlist, e.g. `openai,langchain,wire,http` | -| `FI_DISABLED_INSTRUMENTATIONS` | Empty; denylist wins over allowlist | -| `FI_WIRE_CAPTURE` | `true` | -| `FI_WIRE_MAX_BODY_BYTES` | `4194304`; per request/response text buffering cap | -| `FI_SAMPLE_RATE` | `1`; clamped to [0,1], parent-based root sampling for agent-owned providers | -| `FI_AUTO_DEBUG` | `false`; print masked startup report | -| `FI_CONFIG_FILE` | `./futureagi.toml` | -| `OTEL_SDK_DISABLED` | `true` disables everything silently | - -Example `futureagi.toml`: - -```toml -project_name = "checkout-agent" -project_type = "observe" -instrumentations = ["openai", "langchain", "wire", "http"] -disabled_instrumentations = ["redis"] -wire_capture = true -wire_max_body_bytes = 4194304 -sample_rate = 0.25 -debug = false -hide_inputs = true -``` +| Node ESM or CommonJS | Use `--import` before app imports; CommonJS also supports `--require` | +| Next.js Node runtime | Use guarded `instrumentation.ts` and keep the package external; do not import it into an edge path | +| esbuild, Vite SSR, Rollup or webpack | Opt into the Node-server adapter and deploy the complete agent payload; keep AI SDKs external for library hooks | +| AWS Lambda | Initialize once outside the handler and flush at each invocation boundary, not just process exit | +| Edge / browser | The Node preload is not supported. The `/edge` subpath is a low-level fetch observer, not a configured tracing SDK | +| Existing OpenTelemetry setup | Keep one global provider; configure a single export route for each destination | -This is a flat subset, not full TOML: double-quoted strings, booleans, numbers and -single-line arrays of strings. Comments are supported, including `#` inside quoted -strings. Table declaration lines are ignored. Unknown/unsupported values are ignored. -File keys map to `FI_`; `debug` maps to `FI_AUTO_DEBUG`. +[Framework recipes and packaging requirements](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/FRAMEWORKS.md) -## Doctor, bootstrap and statuses +## Troubleshooting + +Run diagnostics from the app directory with the same FI environment as the app: ```sh -traceai-auto doctor # sends one explicit test export when enabled -traceai-auto doctor --no-check # no test export; reports resolved config/status -traceai-auto bootstrap # list coverage and suggested install commands only -traceai-auto bootstrap --install --package-manager=pnpm # explicit opt-in installation +npx --no-install traceai-auto doctor ``` -Bootstrap defaults to npm when `--package-manager` is omitted; npm/pnpm/yarn are -supported. It never installs target application libraries, only missing instrumentors -for detected targets. Restart the app after installing packages. The installation -branch is implemented but is not exercised against the public registry in tests. - -Reports are JSON, with masked credentials and collector state (`accepted`, `failed`, -or `not_tested`). Startup never probes the collector. With zero sampling, doctor may -have no sampled test span to send; `not_tested` is intentional in that case. - -- `instrumented`: wire hook, server hook, or a library module's patch callback ran. -- `pending`: hooks registered, but no compatible target patch observed yet. -- `not_installed`: no resolvable target library. -- `missing_instrumentor`: target is present but its dedicated package is absent. -- `disabled`: allowlist/denylist, credentials, capture flag, or SDK disable switch. -- `manual`: installed target needs explicit wrapping/callbacks; reason included. -- `failed`: load, registration, patch or known unsupported-version failure; reason included. - -### SDK attribute limits and streaming timing - -In **auto destination mode**, host SDK count/value limits govern native admission. -FI does not clear native attribute storage, re-order it for FI projection, or replay -earlier attributes over later application writes. FI-authored values are protected -before admission; the FI snapshot is built only at export with a late-secret recheck. -Late output or usage dropped by the host is not recovered. Absent optional usage -details are omitted rather than converted to the string `"undefined"`. - -Observed host-dropped attributes, events or links, strings at the configured value -limit (including exact equality), string arrays containing such a value, and -explicit `` / `[truncated]` suffixes trigger lineage-wide withholding. -Every FI content surface becomes ``; `session.id` stays -``, and `traceai.capture.host_data_unknown=true` marks every FI view. -Native SDK dropped counts are retained. Only a closed, typed non-content allowlist -is exempt from withholding, never from secret masking. The decision is sticky for -the native span and identity-retaining mapped/copied FI views, including later views. -There is no partial FI content view or content-derived prefix, length or digest. -Truncated MIME values never establish safe text. Values exported before a loss was -observed are not retracted; already returned snapshots are not mutated retroactively. - -Explicit standalone `FITracer` retains deterministic pre-admission projection -(spec A7), including late library output/usage priority, atomic content groups, -`traceai.capture.dropped_content_count`, and the separate SDK dropped count. -The former `traceai.attribute_projection.metadata` capsule is not produced; older -capsules are still re-sanitized by original keys at export. Host limits are not -raised in either mode. The auto interim is not a guarantee of complete capture or -of retaining canonical output at small host limits. - -`gen_ai.response.time_to_first_chunk` measures the first observed response bytes; -`gen_ai.response.time_to_first_text` separately measures the first nonempty -text/reasoning/tool/audio delta. `traceai.stream.event_arrival_seconds` records event -arrival offsets from request handoff, using byte-boundary timestamps rather than -parse completion time. Content detection includes Cohere tool chunks/reasoning, -nested Mistral content, Responses tool/code deltas, model-requested server/MCP tools -and Gemini executable code. Successful NDJSON EOF accepts a complete final JSON -record without a newline; interrupted or capture-truncated tails still require the -record delimiter, and partial JSON is never promoted to a completed frame. - -## Known limitations - -- **2026-11-05 (Q9 no auto-mode projection; Q8b):** In auto destination mode, - a span whose attributes exceed the host attribute count limit exports the - value-free `` marker for FI content once the host drop - is observed. FI does not project or reconstruct a partial content view. The same - withholding applies to names, status, resource/scope, events and links throughout - the identity-retaining lineage; `session.id` stays `` and every FI view - carries `traceai.capture.host_data_unknown=true`. A closed typed non-content - allowlist retains only its validated values, still subject to secret masking. - Values exported before a loss was observed are not retracted (Q8b residual). - Exact equality with an observed value limit is uncertain too. Native host spans - and unrelated exporters are unchanged by this export-time policy. This is not - complete capture; larger host limits reduce omissions but do not guarantee it. - To keep FI content on spans with many attributes, raise the standard - OpenTelemetry limit `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` (or `OTEL_ATTRIBUTE_COUNT_LIMIT`); - the OpenTelemetry default is 128. -- **2026-10-31 (lossy media context; updated by Q8b):** Observed SDK loss now - withholds all FI content, including ordinary text, base64, embedded/nested JSON, - URL queries and error prose, with the generic unknown marker rather than a - content-derived media-length marker. Malformed JSON tails still fail closed. - Auto records native limit and value-free loss facts before mapping; copied spans - retaining that lineage withhold all non-NON_CONTENT values rather than exporting - a mapper-shortened prefix. Native spans and host exporters - are unchanged by this export policy. Copies without source evidence consult global - SDK limits before environment settings; an unrelated global provider can cause - over-omission. Custom processors that erase identity, limits and loss evidence - can still hide loss. Use `FI_HIDE_INPUTS` / `FI_HIDE_OUTPUTS` / - `FI_HIDE_ERROR_DETAILS` for such routes; no lossless guarantee is made. -- **2026-10-31 (unterminated quoted prose; updated by I1 C3):** Detected SDK - truncation is now withheld before prose parsing. An upstream cut with no marker - or observable SDK limit remains ambiguous: free text with no enclosing JSON, - such as quoted `data`/`mimeType` after `upload failed:`, may lack the closing - delimiter needed by structural recognition. Complete long `data` values retain - catalog omission. Use `FI_HIDE_ERROR_DETAILS` for errors and `FI_HIDE_INPUTS` / - `FI_HIDE_OUTPUTS` for captured input/output prose. -- **2026-10-31 (credential prefixes; updated by I1 C3):** Exact registered-secret - matching does not recognize every unknown prefix. The FI view withholds detected - host-truncated content before that check, including span/event/link detail. - Native foreign spans intentionally keep their original bytes. Upstream/custom - truncation that erases all loss evidence is still not guaranteed detectable; - use FI hide controls and register complete credentials before authored writes. -- **2026-11-04 (option (ii), Mastra manual-only at 11-11):** This tree retains the - manual `@traceai/mastra` package from #567. Use `createFIObservability` or - `createFIMastraExporter`; the registry has no Mastra enable hook and does not - resolve an `/auto` export. The zero-code bridge follows after 11-11. The earlier - unreleased neutral-adapter note and its credential-free/partial-pair zero-code - claims do not describe this tree. No zero-code Mastra support or validation is claimed. -- **2026-10-30 (stream timing bounds):** Timing retains at most 4096 transport byte - boundaries/events. The audio byte-count path retains at most 64 KiB of timing - prefix; missing later event timings are not fabricated. Compressed custom fetch - responses without runtime decoding cannot map compressed offsets onto decoded - events. These cases set `traceai.wire.timing_incomplete`; first-chunk timing remains - available. Response content capture limits and opaque/pipe limitations still apply. -- **2026-11-05 (Q9 supersedes auto-mode A7 small-limit projection):** The host - SDK decides which native attributes it admits; FI does not reorder or reserve - auto-mode slots for output, provenance or usage. Once any host loss is observed, - Q8b withholds FI content for the lineage even when some native content survived. - A default limit such as 128 is not a promise that output survives a larger span. - Explicit standalone `FITracer` projection is separate and remains supported. - -Dated **2026-10-30**. These are coverage/environment limitations, not claims of -universal instrumentation support. - -- **2026-10-30 (S6 recognition):** Media and credential recognition follow the shared - `privacy.json` catalog, not semantic inference. With complete SDK context, unknown, - non-magic base64 outside its payload fields is not classified as media. Exact scalar data URIs own their - complete value; URIs inside prose are delimiter-scanned. Catalog depth/node/text - exhaustion fails closed, retaining span identity/timing rather than partial content. - Use the `FI_HIDE_INPUTS`, `FI_HIDE_OUTPUTS` and `FI_HIDE_ERROR_DETAILS` controls - when application content requires stronger protection than structural recognition. -- **2026-10-30 (configuration):** Config/profile parsing supports flat scalar values - and string arrays, plus named `[profiles.name]` sections in `.fi/config.toml` and - `$XDG_CONFIG_HOME/fi/config.toml` (default `~/.config/fi/config.toml`). Multiline - TOML and arbitrary TOML expressions are not supported. Credential pairs resolve - atomically: explicit arguments, environment, `futureagi.toml`, fi project profile, - then fi user profile; incomplete sources are ignored with a value-free warning - naming the missing half. Ignored FI halves from every source still seed S5 and - cannot appear in diagnostics. `FI_PROFILE`/the `profile` option selects the profile. -- **2026-10-30 (ROUND5 B2 scope; zero-code/one-line init):** Credentials that never cross an instrumented - transport and have no credential marker cannot be inferred from free text. Opaque - `Request`/stream bodies are never parsed or buffered; a body-only credential echoed - as plain text can therefore remain unknown when it is neither an environment/FI - value nor explicitly registered. This includes `authorization_token` inside a native - `Request` or `ReadableStream` body; the original credential-bearing application 401 - remains unchanged. Those unparsed fields are outside ROUND5 B2's parsed-body scope. - Materialized `URLSearchParams` and native FormData text entries are harvested - before capture limits, including duplicate fields; opaque pre-encoded form strings - are not form-decoded. Suppressed calls observe headers/URL only, never bodies. - Application transformations (signature extraction, JSON encoded inside - strings, numeric re-typing, base64 or splitting) are not a general secret-detection - guarantee. Use `registerSecret` for additional exact values and the `FI_HIDE_INPUTS`, - `FI_HIDE_OUTPUTS`, `FI_HIDE_INPUT_TEXT`, `FI_HIDE_OUTPUT_TEXT` and - `FI_HIDE_ERROR_DETAILS` controls for sensitive content/error surfaces. -- **2026-10-30:** One-line/manual instrumentation with transport hooks disabled has - only initialization FI/environment seeds and explicit registrations; credentials - never observed by those hooks are not automatically known. The bounded LRU can - evict old values after 1024 distinct credentials; re-register long-lived values - when reused. The same hide controls above remain available. -- **2026-10-30:** Transport credential-discovery exhaustion (more than 1024 distinct - values in one observation, excessive nesting/nodes/text or an already-materialized - body over the 4 MiB discovery cap) fails closed process-wide: subsequent FI spans - keep identity/timing but drop content on every surface. Application requests, - responses and other exporters are unchanged. This latch lasts until process restart; - it cannot safely be reset while unknown credentials could still be echoed. Ordinary - credentials longer than 8192 characters are supported within the aggregate budgets. -- **2026-10-30:** Unnamed in-memory esbuild ESM output (`write:false` without - `outfile`/`outdir`) is left byte-identical to the no-plugin build: there is no - deployable sibling path for the ESM bootstrap. Use an output path or external - `node --require` preload for instrumentation. Unnamed CJS/default outputs receive - their preload in memory, with no filesystem output access. Custom JavaScript output - extensions are supported for named CJS and ESM outputs. -- **2026-10-30 (zero-code/one-line init):** Cancelling an unused clone can finalize the shared capture before the original - branch is consumed. Cancelling a released reader can likewise finalize capture - even when the native cancellation promise rejects. Application bytes, promises - and errors remain unchanged, but the one span can be incomplete, omit later output/ - usage and end too early. These two cases were independently reproduced even when - later consumption returns the complete answer and usage. Branch-aware cancellation - finalization is deferred. -- **2026-10-30 (zero-code/one-line init):** A provider failure carried inside an HTTP 200 stream (for example OpenAI Responses - `response.failed`) can retain HTTP-derived OK status without the provider error - description, even when `gen_ai.response.finish_reasons` records `failed`. HTTP - success is not currently overridden by that in-stream provider failure. -- Native `pipeTo`/`pipeThrough` algorithms bypass instance reader methods. These - paths retain metadata-only completion (`traceai.wire.body_unavailable=true`), - without replacing the stream/sink or fabricating output bytes. Their completion - timing is the pipe/downstream completion, not a byte-level transport timestamp. - Directly borrowing prototype methods, or freezing/non-configurably replacing - custom stream methods, can bypass observation entirely. Native Response helpers - and ordinary readers/iterators/clone/tee have byte-level regression coverage. -- Custom lazy/non-native thenables are not a validated Node fetch implementation; - the Node observer does not invoke their overridden `then` to start them. Node's - observer uses V8 promise-parent notifications and native rejection mirrors to - preserve unhandled/handled rejection events and default fatal exits for ordinary - fetch/read promises. Pre-created custom promises consumed before agent startup - are not fully observable. The Node-free edge adapter lacks that Node-specific - rejection-lifecycle mechanism and is not a drop-in process agent. -- An SDK without a processor-removal API can retain one inert agent processor after - shutdown. Agent callbacks stop, application processors remain active, and init does - not add another processor. Optional enable entries own their integration's hook - cleanup when they do not return a disable handle. -- Covered ancestry traversal is bounded to 256 wrapper/native ancestors; esbuild glob - expansion is bounded to 10,000 filesystem entries and 64 recursive directory levels. - -- No outgoing **wire spans** for `node:http`/`node:https`, direct `undici.fetch`/ - `undici.request`, custom fetch implementations captured before preload, WebSocket, - native gRPC, worker processes, browsers or arbitrary non-fetch transports. HTTP/ - Undici diagnostics do harvest credentials independently of span creation. Preload - each separate Node process and worker isolate (globals/registries are not shared - across V8 isolates). OpenAI's included dedicated instrumentation covers its - supported default transport; the global fetch wire hook does not cover node-fetch. -- URL matching is limited to the catalog. Restricted-provider endpoints on a custom - proxy host may not match. No speculative interception of unknown AI protocols. -- Request objects and opaque/pre-encoded multipart streams are never cloned/read; - known provider endpoints retain metadata/body-unavailable capture only. Use native - FormData for text fields and file metadata. Binary request sizes are included only - when the body or declared content length exposes the size. -- A response must be consumed or cancelled to finish capture. Unread or never-ending - responses do not yield completed spans. Force-flush never waits for unread responses. -- Bedrock binary event-stream responses record request/status only. Audio streaming - and base64 variants count bytes but do not reconstruct audio/timestamps. No audio - bytes are exported. -- Real OpenAI calls, real Agents processor coexistence and real Cursor class ownership - are validated here; registry completeness does not imply every one of the 45 - packages works with every version. Optional enable-entry registry tests validate - the adapter contract, not each external framework deployment. Instrumentations - depending on obsolete module paths may remain pending. +Doctor sends one test span when instrumentation is enabled. Check the JSON `collector` field: `accepted`, `failed` or `not_tested (...)`. **The CLI exit code alone is not a collector-health check.** Doctor is a separate process, so its `pending` library hooks do not tell you whether your running app has imported that library. -## Legal notices +For a configuration/status report without the test export: -The published package includes `dist/THIRD-PARTY-NOTICES`, with the full installed -LICENSE, LICENCE, COPYING and NOTICE files for third-party inputs to all four -launcher/edge bundles. `dist/third-party-manifest.json` maps each bundle to its -third-party package names and versions; first-party `@traceai/*` packages are listed -separately in the notices. These files supplement, and do not replace, the existing -linked `.LEGAL.txt` output. Keep them with any redistributed launcher or edge payload. -The build fails if a bundled third-party package has no readable, nonempty license -text file; a license identifier or linked source comment alone is not sufficient. +```sh +npx --no-install traceai-auto doctor --no-check +``` -## Development and verification +| Symptom | What to check | +| --- | --- | +| Everything is `disabled` | Supply both FI keys in the same source, or configure an application destination. Check `OTEL_SDK_DISABLED` and the allowlist/denylist. | +| `collector: failed` | Check `FI_BASE_URL`, both keys, outbound connectivity and collector access. Initialization itself never probes the collector. | +| `collector: not_tested` | No completed test export yet, `--no-check`, zero sampling, or an application-owned destination. It is not a success signal. | +| `missing_instrumentor` | Install the suggested dedicated package from bootstrap, then restart. | +| Library stays `pending` | Load the agent before the SDK; check supported versions and keep the SDK external when bundling. A registered hook is not proof a call was traced. | +| Library reports `manual` | Use its explicit wrapper/processor API; automatic framework-level tracing is unavailable for that installed version. | +| No wire span | Consume or cancel the response; check the catalog and transport. Direct `undici.fetch`, `node-fetch`, WebSocket and gRPC are not global-fetch wire capture. | +| Output is missing or unknown | Check hide flags, body cap and host OTel attribute limits. Dropped/truncated host data cannot be reconstructed. | +| Duplicate spans | Remove duplicate manual instrumentation/preloads; avoid layering an invocation wrapper over another Lambda tracing layer. | + +## Limits to plan for + +- **Per process/isolate:** preload every Node worker or child process you want to observe; globals are not shared between isolates. +- **Bounded capture:** the default text body cap is 4 MiB. Request streams and opaque `Request` bodies are not consumed for inspection. Native `pipeTo`/`pipeThrough` can provide metadata-only completion. +- **Response-driven completion:** unread and never-ending responses do not produce completed wire spans. Flush does not consume them for you. Cancelling a clone can end the shared capture before another branch finishes. +- **Provider error semantics:** an error inside an HTTP 200 stream can retain HTTP-derived OK status; inspect finish reasons as well. +- **Best-effort delivery:** natural-exit/SIGTERM flushes are bounded. Hard termination, Lambda freezing and collector failures can lose spans. The agent never makes app success depend on export success. + +[Architecture, complete capture limits and redistribution notices](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/ARCHITECTURE.md) -From `typescript/`: +## Legal notices -```sh -pnpm install --ignore-scripts -pnpm run build:deps -pnpm --filter @traceai/auto run build # TS references build OpenAI and Anthropic CJS prerequisites -pnpm --filter @traceai/auto test -pnpm --filter @traceai/fi-core test --runInBand -``` +Keep `dist/THIRD-PARTY-NOTICES`, `dist/third-party-manifest.json` and the linked `.LEGAL.txt` files with any redistributed launcher or edge payload. [Redistribution details](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/traceai_auto/docs/ARCHITECTURE.md#legal-notices). + +## More resources -The agent builder requires the resolved OpenAI instrumentor to appear in esbuild's -input graph and fails the build if it is missing; it cannot silently produce a -wire-only launcher after an `--ignore-scripts` install. - -Tests load **all 4,985 shared S6 privacy fixtures** and discover the wire goldens -**recursively** under `spec/auto-instrumentation/fixtures/` (340 on 2026-11-01: 47 -baseline + 293 round6 A8/A10 files, replayed with their `chunk_sizes`, -`error_after_chunks` and per-chunk `arrival_offsets_ms`; the ten decoded-payload -Bedrock files 157-166 go to the A10 recognizer, not the wire endpoint, per the spec), -including unavailable requests, encoded gzip responses and mid-stream failures; they -assert names/status descriptions/exact attribute subsets, absence/presence and -`not_contains` privacy expectations, and compare catalog bytes. Local mock-server e2e tests spawn real Node `--import`, `-r`, and -NODE_OPTIONS entry points, using a local OTLP collector rather than Future AGI or a -real AI provider. Other tests cover body caps, nesting/suppression, binary/BYOB/audio, -configuration precedence, privacy, global SDK coexistence and incoming transactions. +[Future AGI tracing guide](https://docs.futureagi.com/docs/tracing) · [Integration guides](https://docs.futureagi.com/docs/integrations/traceai) · [Manual TypeScript SDK](https://github.com/future-agi/traceAI/blob/dev/typescript/packages/fi-core/README.md) · [Report an issue](https://github.com/future-agi/traceAI/issues) diff --git a/typescript/packages/traceai_auto/docs/ARCHITECTURE.md b/typescript/packages/traceai_auto/docs/ARCHITECTURE.md new file mode 100644 index 000000000..0ad7e2786 --- /dev/null +++ b/typescript/packages/traceai_auto/docs/ARCHITECTURE.md @@ -0,0 +1,474 @@ +# Capture architecture + +[Quick start](../README.md) · [Configuration](CONFIGURATION.md) · [Framework recipes](FRAMEWORKS.md) + +This reference describes the Node agent’s capture, privacy and lifecycle boundaries. For setup, start with the quick start. + +## Initialization and ownership + +`init(options)` configures the process **once**. It retains the existing provider, +configuration and hooks on every later call, logging one diagnostic line. It returns +the **same tracer provider object**, or `undefined` when initialization is disabled or +failed. Diagnostics are returned by `report()`, not `init()`. Other exports are +`forceFlush(): Promise`, `shutdown(): Promise`, and +`registerSecret(value: unknown): void` (also exported additively by `@traceai/fi-core`). +Integrations can register credentials observed outside a transport without supplying +span context or retaining per-trace credential groups. Non-string values and strings +shorter than eight characters are ignored; registration never coerces user objects. +Shutdown removes this agent's hooks/processors but never shuts down an +application-owned global OTel provider. Reinitialization after shutdown or failed +startup, changing privacy/exporter settings, and disabling an enabled integration +by calling `init()` again are unsupported: restart the process to change configuration. +For short-lived programs, finish reading responses, then `await forceFlush()` or +`await shutdown()`. + +The preload does not perform network I/O. The first real completed span initiates +a bounded batch export, or `doctor` can explicitly send a test span. FI credentials +are required only for the default FI destination. `init({ exporter })`, +`init({ spanProcessor })`, `init({ provider })`, or an existing application OTel provider +activates instrumentation without FI credentials and without creating an FI export route. +With neither a destination nor a complete FI pair, both layers are disabled with one warning. +FI project routing never replaces application `service.name` or environment resource +identity. Export/integration failures warn once per +failure category, do not log credentials/provider bodies, and leave the app running. +A one-shot `beforeExit` handler flushes completed spans. The agent adds no SIGINT +handler. Its SIGTERM listener starts a bounded flush and preserves app-owned listeners; +without one, it restores the default action and re-raises SIGTERM (preserving the signal +exit). Explicit flush/shutdown have a five-second deadline. Shutdown reserves up to +2.5 seconds for flush and then invokes exporter shutdown with the remaining 2.5 seconds, +even if flush never resolves. A failed initialization rolls back this agent's hooks, +processor and newly installed globals; app-owned globals are retained. The first +configuration remains terminal, so a later `init()` cannot retry or rewire the +process. SIGKILL/forced termination cannot guarantee delivery. + +## Capture layers + +### Wire layer + +The runtime reads a byte-identical packaged copy of the shared +`spec/auto-instrumentation/wire-protocols.json` catalog. It patches `globalThis.fetch`, +preserving the fetch function's name, length and custom properties via a Proxy. + +Recognized formats include: + +- OpenAI-compatible chat/completions, Responses, embeddings, image generation, + moderation, transcription/translation and speech generation; compatible gateways + and providers are recognized by the catalog. Unknown hosts use the endpoint's + protocol-specific fallback (`openai_compatible`, `anthropic_compatible`, + `gemini_compatible`, or `cohere_compatible`). +- Anthropic messages, Gemini/Vertex generate-content and embeddings, Cohere chat, + embed and rerank, Bedrock converse/invoke, Ollama chat/generate/embed. +- Deepgram listen/speak, ElevenLabs TTS/STT, AssemblyAI transcript submission and + completed polling, and Cartesia bytes/SSE TTS. + +Request-body validation prevents unrelated application endpoints from being claimed. +Request streams, opaque Request objects and over-limit bodies are captured **only for +catalog providers**, without request content and with `traceai.wire.body_unavailable` +or `traceai.wire.truncated`. Unknown proxies need a validated materialized request. +The actual fetch is invoked before inspecting materialized bodies: no request cloning, +stream read-ahead or request-side wait is added. + +Wire spans are OTel CLIENT spans with the saved parent context. Start time is the +transport handoff, before URL validation or request inspection; end time is the last +observed byte/error, captured **before** copying, decoding or parsing. Parsing is +excluded from duration. Streams are observed by tapping the existing readers and +iterators, not by replacing the body. The caller receives the **original fetch +Promise and original Response**, including subclass/helper methods and native headers, +URL/type, clone, tee, BYOB support (or its original rejection), cancellation promises, +and chunk identities. Capture copies bytes into its own bounded buffers without +transferring/detaching app storage. No read, reader lock, synthetic Response, automatic +clone or eager tee is created for an unread body. Export is never awaited between +chunks; reading multiple clone/tee branches produces at most one wire span. + +Text/JSON/SSE/NDJSON capture stops at the configured byte cap, discards buffered +content, and sets `traceai.wire.truncated`. Audio/binary responses are counted, not +buffered. JSON/SSE TTS base64 audio is counted with a bounded streaming scanner, +without accumulating or decoding audio. If a custom fetch exposes compressed JSON/SSE +TTS bytes, the compressed capture is bounded by the body cap and decoded asynchronously +after consumption; decoded chunks go straight to that scanner, not a retained audio +buffer. The decoded-work cap also applies; exceeding it sets `traceai.wire.truncated`. +FormData text fields are read independently +of upload size, each capped at 64 KiB; file bytes are never read, and only their +size/MIME/filename are retained. Catalog embedding formats record dimensions, not +vectors. Use `traceConfig.hideEmbeddingVectors` (or `FI_HIDE_EMBEDDING_VECTORS=true`) +to also hide vectors in generic JSON fallbacks and forwarded library/application spans. + +Native fetch normally decompresses responses. When a custom fetch exposes encoded +bytes, bounded asynchronous Node decoders handle gzip/deflate/Brotli before parsing, +plus zstd when Node exposes it. The Node-free layer uses Web `DecompressionStream` +for the codecs its runtime supports. Unavailable decoders +keep request-side metadata and set `traceai.wire.undecoded_encoding`. Already-decoded +JSON/SSE is not decoded twice. A transport failure mid-stream is ERROR with parsed +partial output; consumer cancellation is OK with `traceai.wire.incomplete = true`. + +Wire **span creation** is skipped for OTel-suppressed contexts, nested captures, and +LLM/EMBEDDING/RERANKER/GUARDRAIL ancestors, even through intermediate native SDK +spans. Outgoing credential observation is never skipped by these rules: fetch reads +headers/URL without inspecting suppressed request bodies. Node HTTP and Undici +diagnostic-channel subscribers also observe outgoing headers/URLs (including direct, +cached clients and non-AI routes); they do not create additional wire spans. The FI processor keeps weak live ancestry, so a model kind set after span +creation is also respected. Both `gen_ai.span.kind` and older `fi.span.kind` are +understood, including FI-wrapped spans. Uncovered CHAIN/AGENT/TOOL spans remain +eligible parents. Session, user and other fi-core context attributes are propagated. + +Wire attributes pass through fi-core's TraceConfig. Additional conservative handling +covers flattened message keys, audio transcripts, and raw body copies when content +hiding is enabled. `FI_HIDE_INPUTS`, `FI_HIDE_OUTPUTS`, `FI_HIDE_INPUT_MESSAGES`, +`FI_HIDE_OUTPUT_MESSAGES`, `FI_HIDE_INPUT_TEXT`, `FI_HIDE_OUTPUT_TEXT`, +`FI_HIDE_INPUT_IMAGES`, `FI_HIDE_ERROR_DETAILS`, and `FI_PII_REDACTION` are supported. +TTS `gen_ai.audio.transcript` obeys input-hide flags; STT obeys output-hide flags. +Hidden transcripts use `[REDACTED]`, exactly like hidden input/output values. + +### Privacy + +One export-time boundary sanitizes **every span sent through the agent's exporter**: +wire spans, dedicated-library spans and forwarded application spans. It builds a +detached snapshot of the ended native span; other exporters keep the original +span/resource/context unchanged by that export policy. In auto destination mode, +foreign attributes are never rewritten by FI policy. FI-authored attributes still +receive write-time hide flags, image caps, PII and known-secret protection, so +other exporters never receive authored content that those protections hid. +Application active context uses the native span; instrumentor writes use a separate +FI wrapper. Explicit standalone `FITracer` retains its active-wrapper behavior. +This is an **interim export-built view**, not complete capture: data the host SDK +already dropped or truncated may be omitted or marked unknown, never reconstructed. +Attributes (arrays element-wise), status descriptions, event names/attributes, +link attributes/contexts, resources, instrumentation scope name/version/attributes +and tracestate values all pass through this boundary. Nonfinite numbers become +`"NaN"`, `"Infinity"` or `"-Infinity"`. Unsupported attribute objects become type +names; user attribute getters, `toString` and `toJSON` are never called. The standard +ReadableSpan context method is supported for library-generated snapshots. + +Work is bounded across each snapshot (20,000 values, 8 MiB text, 32 array levels; +credential discovery also has depth/count limits). On a sanitizer error, content, +status descriptions, events, links, scope and resource content are dropped for that +span while its identity and timing survive. Upstream wire scrubbing is best-effort, +not the security boundary. Before TraceConfig masking, every recorded string is +scrubbed: + +- Data URLs and long base64 media fields (`data`, `bytes`, `b64_json`, `file_data`, + `audio_base64`, `images` elements) become ``. Image-generation + `b64_json` uses the format contract's `` placeholder. Raw audio is + never retained, even for tiny inline audio clips: scalar/array `audio` and + `input_audio` JSON leaves, indexed flattened fields and `inlineData.data.0` with + a sibling audio MIME type all use the same no-minimum-size audio policy. + MIME metadata and transcripts remain subject to their own content-hide controls. +- Credential-named keys are replaced with `` recursively, including + `authorization_token` in MCP server settings. The only logprob-name exception + is the explicitly bound provider-root policy below; access-key identifiers + remain masked. Catalog credential shapes are checked independently of names. +- Signed URL parameters, including AWS/GCP signatures and Azure SAS fields, have their + values replaced with ``, including percent-encoded parameter names. +- A single `globalThis[Symbol.for('traceai.secrets')]` registry is shared by installed, + embedded, CJS and ESM package copies in the process/isolate. It retains at most 1024 + values of eight or more characters, evicting the least recently registered/observed + value. Immutable longest-first snapshots are cached until membership changes. At + initialization it is seeded with configured FI keys and credential-like environment + values (names containing KEY, TOKEN, SECRET, PASSWORD, PASSWD, CREDENTIAL or AUTH). + Transport-observed header/query values, parsed credential fields and explicit + `registerSecret` calls augment it. Every exported span, including application-owned + instrumentation spans and unrelated/later traces, uses the current registry at + export time; there is no span-start credential inheritance or SDK-state inspection. + Credential headers and export-boundary credential fields use one shared list, + including `x-goog-api-key`, `anthropic-api-key` and `ocp-apim-subscription-key`. + Values never appear in diagnostics, reports or error messages. Every registered + value is removed from every exported string, including names, status, events, + links, resource/scope attributes and tracestate; other exporters remain unchanged. + Materialized bodies discarded by the capture cap, URLSearchParams values and + uncropped/duplicate FormData text fields still undergo bounded credential discovery + before disposal. File bytes are never inspected. Credential-discovery budget + exhaustion or failure while inspecting a materialized body drops all exported + content, not span identity, timing or status code; this also omits the truncation + marker on that fail-closed snapshot. Opaque, unparsed bodies instead follow the + scoped limitation below; their observable headers/URL still augment the registry. + Configured FI keys remain protected by the export policy even after LRU churn. + +For the closed, versioned `logprob_tokens` catalog paths, **logprob tokens are checked against registered secrets and credential shapes, not key-name rules**. +This exception is available only from the parsed OpenAI chat response root at the +instrumentation handoff: `choices[*].logprobs.content[*].token` and +`choices[*].logprobs.content[*].top_logprobs[*].token`. It does not apply to nested +JSON in user content, tool output, arbitrary attributes, or flattened key names. +If value masking changes any choice's `message.content` or `delta.content`, its +**entire logprobs block** is masked, including alternatives and byte arrays. +Buffered streaming responses are scanned by choice index before export; once a +choice is masked, earlier buffered blocks and all later blocks for that choice +are masked too. Concatenated deltas are checked for split secrets, and affected +delta text is masked. A masked individual token also loses its byte-array alias. +The staged OpenAI instrumentor exports the buffered response once, not each delta. +Other provider structures have no new exception without catalog/source evidence. + +Hide policies match semantic families, including `input.*`, `output.*`, +`gen_ai.input.*`, `gen_ai.output.*`, system instructions, tools, image/file URIs and +side-specific audio transcripts. `FI_HIDE_LLM_INVOCATION_PARAMETERS=true` additionally +hides `gen_ai.request.parameters`, explicit invocation parameters and tool schemas. +`hideEmbeddingVectors` also covers JSON fallbacks such as generic Bedrock invoke, +not only spans classified as EMBEDDING. Image hiding conservatively hides raw input +and input-message families to prevent URI aliases escaping the image policy. + +Do not put arbitrary secrets in prompts: unknown free-text secrets cannot always be +recognized. These policies cover dedicated-instrumentor spans only when they pass +through **this agent's** exporter; unrelated exporters retain their own policy. + +## Library hooks and existing providers + +Target/instrumentation resolution is from the application's working directory. +Install packages in that app and run from its directory. Hook registration is eager; +target patching occurs on import/require. The import-in-the-middle loader is registered +before `registerInstrumentations`, restricted to target entry points rather than +all ESM internals (which can break SDK shim live bindings). Node 26 currently emits +a deprecation warning for upstream `module.register()` when library hooks are enabled. +OpenAI v4/node-fetch can additionally emit +Node's upstream `punycode` deprecation. Neither warning is suppressed by the agent. +Disabled integrations are never loaded or registered, including integrations sharing +a target (for example DeepSeek and OpenAI). + +Mastra is manual-only in this release. Use `createFIObservability` or `createFIMastraExporter`; the registry does not load a Mastra auto entry. + +For Genkit and Vercel AI SDK, the registry calls an installed `enable` +entry (also accepting the integration's named enable function). It prefers the +optional `/auto` subpath. Versions without that entry are reported `manual`, not +forced into an incompatible constructor; a broken entry fails open. The entry +receives `{ tracerProvider, spanProcessor, traceConfig }` for the selected agent +branch; older no-argument entries can ignore these optional arguments and use the +initialized agent. An instrumentor returned by the entry is explicitly bound to +that provider, not an unrelated global provider. This does not replace any global +provider or synthesize FI credentials. +Manual-only integrations include Claude Agent SDK (`wrapQuery`), Codex SDK +(`wrapCodex`/`manuallyInstrument`), Pi Coding Agent (session wrapper), VoltAgent +(processor), ElevenLabs (client wrapper), Google GenAI SDK (`instrumentGoogleGenAI`) +and Hugging Face Transformers (module/pipeline wrapper). Their underlying fetch AI calls can still be traced. +Existing dedicated instrumentations retain their supported-version and method +coverage constraints; registration does not expand those APIs. OpenAI Agents uses +additive processor registration (`exclusiveProcessor:false`), preserving application +processors. Shutdown recognizes processor/class-registry ownership as well as +shimmer wrappers, and leaves application-owned patches intact. + +If `@opentelemetry/instrumentation-http` is installed and resolvable in the app, it is enabled for incoming SERVER spans only. Outgoing non-AI HTTP spans are not enabled by default. + +### Existing OTel SDKs + +An existing global SDK provider is retained, with the FI processor added beside its +existing processors. Its sampler/context manager remain authoritative. OTel JS 2.x +has no public late-add-processor API, so this uses the live MultiSpanProcessor list; +the public 1.x API is also recognized. FI project resource attributes are enriched +only on the FI export copy. If a custom provider does not expose either supported +attachment mechanism, the agent warns and uses an isolated AI provider without +replacing the global provider. This is a degraded fallback, not a promise of +processor attachment to arbitrary OTel implementations. + +## SDK attribute limits and streaming timing + +In **auto destination mode**, host SDK count/value limits govern native admission. +FI does not clear native attribute storage, re-order it for FI projection, or replay +earlier attributes over later application writes. FI-authored values are protected +before admission; the FI snapshot is built only at export with a late-secret recheck. +Late output or usage dropped by the host is not recovered. Absent optional usage +details are omitted rather than converted to the string `"undefined"`. + +Observed host-dropped attributes, events or links, strings at the configured value +limit (including exact equality), string arrays containing such a value, and +explicit `` / `[truncated]` suffixes trigger lineage-wide withholding. +Every FI content surface becomes ``; `session.id` stays +``, and `traceai.capture.host_data_unknown=true` marks every FI view. +Native SDK dropped counts are retained. Only a closed, typed non-content allowlist +is exempt from withholding, never from secret masking. The decision is sticky for +the native span and identity-retaining mapped/copied FI views, including later views. +There is no partial FI content view or content-derived prefix, length or digest. +Truncated MIME values never establish safe text. Values exported before a loss was +observed are not retracted; already returned snapshots are not mutated retroactively. + +Explicit standalone `FITracer` retains deterministic pre-admission projection +(spec A7), including late library output/usage priority, atomic content groups, +`traceai.capture.dropped_content_count`, and the separate SDK dropped count. +The former `traceai.attribute_projection.metadata` capsule is not produced; older +capsules are still re-sanitized by original keys at export. Host limits are not +raised in either mode. The auto interim is not a guarantee of complete capture or +of retaining canonical output at small host limits. + +`gen_ai.response.time_to_first_chunk` measures the first observed response bytes; +`gen_ai.response.time_to_first_text` separately measures the first nonempty +text/reasoning/tool/audio delta. `traceai.stream.event_arrival_seconds` records event +arrival offsets from request handoff, using byte-boundary timestamps rather than +parse completion time. Content detection includes Cohere tool chunks/reasoning, +nested Mistral content, Responses tool/code deltas, model-requested server/MCP tools +and Gemini executable code. Successful NDJSON EOF accepts a complete final JSON +record without a newline; interrupted or capture-truncated tails still require the +record delimiter, and partial JSON is never promoted to a completed frame. + +## Known limitations + +- In auto destination mode, + a span whose attributes exceed the host attribute count limit exports the + value-free `` marker for FI content once the host drop + is observed. FI does not project or reconstruct a partial content view. The same + withholding applies to names, status, resource/scope, events and links throughout + the identity-retaining lineage; `session.id` stays `` and every FI view + carries `traceai.capture.host_data_unknown=true`. A closed typed non-content + allowlist retains only its validated values, still subject to secret masking. + Values exported before a loss was observed are not retracted. + Exact equality with an observed value limit is uncertain too. Native host spans + and unrelated exporters are unchanged by this export-time policy. This is not + complete capture; larger host limits reduce omissions but do not guarantee it. + To keep FI content on spans with many attributes, raise the standard + OpenTelemetry limit `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` (or `OTEL_ATTRIBUTE_COUNT_LIMIT`); + the OpenTelemetry default is 128. +- Observed SDK loss now + withholds all FI content, including ordinary text, base64, embedded/nested JSON, + URL queries and error prose, with the generic unknown marker rather than a + content-derived media-length marker. Malformed JSON tails still fail closed. + Auto records native limit and value-free loss facts before mapping; copied spans + retaining that lineage withhold all non-NON_CONTENT values rather than exporting + a mapper-shortened prefix. Native spans and host exporters + are unchanged by this export policy. Copies without source evidence consult global + SDK limits before environment settings; an unrelated global provider can cause + over-omission. Custom processors that erase identity, limits and loss evidence + can still hide loss. Use `FI_HIDE_INPUTS` / `FI_HIDE_OUTPUTS` / + `FI_HIDE_ERROR_DETAILS` for such routes; no lossless guarantee is made. +- Detected SDK + truncation is now withheld before prose parsing. An upstream cut with no marker + or observable SDK limit remains ambiguous: free text with no enclosing JSON, + such as quoted `data`/`mimeType` after `upload failed:`, may lack the closing + delimiter needed by structural recognition. Complete long `data` values retain + catalog omission. Use `FI_HIDE_ERROR_DETAILS` for errors and `FI_HIDE_INPUTS` / + `FI_HIDE_OUTPUTS` for captured input/output prose. +- Exact registered-secret + matching does not recognize every unknown prefix. The FI view withholds detected + host-truncated content before that check, including span/event/link detail. + Native foreign spans intentionally keep their original bytes. Upstream/custom + truncation that erases all loss evidence is still not guaranteed detectable; + use FI hide controls and register complete credentials before authored writes. +- This tree retains the + manual `@traceai/mastra` package. Use `createFIObservability` or + `createFIMastraExporter`; the registry has no Mastra enable hook and does not + resolve an `/auto` export. No zero-code Mastra support is provided by this registry. +- Timing retains at most 4096 transport byte + boundaries/events. The audio byte-count path retains at most 64 KiB of timing + prefix; missing later event timings are not fabricated. Compressed custom fetch + responses without runtime decoding cannot map compressed offsets onto decoded + events. These cases set `traceai.wire.timing_incomplete`; first-chunk timing remains + available. Response content capture limits and opaque/pipe limitations still apply. +- The host + SDK decides which native attributes it admits; FI does not reorder or reserve + auto-mode slots for output, provenance or usage. Once any host loss is observed, + The export boundary withholds FI content for the lineage even when some native content survived. + A default limit such as 128 is not a promise that output survives a larger span. + Explicit standalone `FITracer` projection is separate and remains supported. + +- Media and credential recognition follow the shared + `privacy.json` catalog, not semantic inference. With complete SDK context, unknown, + non-magic base64 outside its payload fields is not classified as media. Exact scalar data URIs own their + complete value; URIs inside prose are delimiter-scanned. Catalog depth/node/text + exhaustion fails closed, retaining span identity/timing rather than partial content. + Use the `FI_HIDE_INPUTS`, `FI_HIDE_OUTPUTS` and `FI_HIDE_ERROR_DETAILS` controls + when application content requires stronger protection than structural recognition. +- Config/profile parsing supports flat scalar values + and string arrays, plus named `[profiles.name]` sections in `.fi/config.toml` and + `$XDG_CONFIG_HOME/fi/config.toml` (default `~/.config/fi/config.toml`). Multiline + TOML and arbitrary TOML expressions are not supported. Credential pairs resolve + atomically: explicit arguments, environment, `futureagi.toml`, fi project profile, + then fi user profile; incomplete sources are ignored with a value-free warning + naming the missing half. Ignored FI halves from every source still seed the secret registry and + cannot appear in diagnostics. `FI_PROFILE`/the `profile` option selects the profile. +- Credentials that never cross an instrumented + transport and have no credential marker cannot be inferred from free text. Opaque + `Request`/stream bodies are never parsed or buffered; a body-only credential echoed + as plain text can therefore remain unknown when it is neither an environment/FI + value nor explicitly registered. This includes `authorization_token` inside a native + `Request` or `ReadableStream` body; the original credential-bearing application 401 + remains unchanged. Those unparsed fields are outside parsed-body credential discovery. + Materialized `URLSearchParams` and native FormData text entries are harvested + before capture limits, including duplicate fields; opaque pre-encoded form strings + are not form-decoded. Suppressed calls observe headers/URL only, never bodies. + Application transformations (signature extraction, JSON encoded inside + strings, numeric re-typing, base64 or splitting) are not a general secret-detection + guarantee. Use `registerSecret` for additional exact values and the `FI_HIDE_INPUTS`, + `FI_HIDE_OUTPUTS`, `FI_HIDE_INPUT_TEXT`, `FI_HIDE_OUTPUT_TEXT` and + `FI_HIDE_ERROR_DETAILS` controls for sensitive content/error surfaces. +- One-line/manual instrumentation with transport hooks disabled has + only initialization FI/environment seeds and explicit registrations; credentials + never observed by those hooks are not automatically known. The bounded LRU can + evict old values after 1024 distinct credentials; re-register long-lived values + when reused. The same hide controls above remain available. +- Transport credential-discovery exhaustion (more than 1024 distinct + values in one observation, excessive nesting/nodes/text or an already-materialized + body over the 4 MiB discovery cap) fails closed process-wide: subsequent FI spans + keep identity/timing but drop content on every surface. Application requests, + responses and other exporters are unchanged. This latch lasts until process restart; + it cannot safely be reset while unknown credentials could still be echoed. Ordinary + credentials longer than 8192 characters are supported within the aggregate budgets. +- Unnamed in-memory esbuild ESM output (`write:false` without + `outfile`/`outdir`) is left byte-identical to the no-plugin build: there is no + deployable sibling path for the ESM bootstrap. Use an output path or external + `node --require` preload for instrumentation. Unnamed CJS/default outputs receive + their preload in memory, with no filesystem output access. Custom JavaScript output + extensions are supported for named CJS and ESM outputs. +- Cancelling an unused clone can finalize the shared capture before the original + branch is consumed. Cancelling a released reader can likewise finalize capture + even when the native cancellation promise rejects. Application bytes, promises + and errors remain unchanged, but the one span can be incomplete, omit later output/ + usage and end too early. Later consumption can still return the complete answer and usage; branch-aware cancellation finalization is not provided. +- A provider failure carried inside an HTTP 200 stream (for example OpenAI Responses + `response.failed`) can retain HTTP-derived OK status without the provider error + description, even when `gen_ai.response.finish_reasons` records `failed`. HTTP + success is not currently overridden by that in-stream provider failure. +- Native `pipeTo`/`pipeThrough` algorithms bypass instance reader methods. These + paths retain metadata-only completion (`traceai.wire.body_unavailable=true`), + without replacing the stream/sink or fabricating output bytes. Their completion + timing is the pipe/downstream completion, not a byte-level transport timestamp. + Directly borrowing prototype methods, or freezing/non-configurably replacing + custom stream methods, can bypass observation entirely. Native Response helpers + and ordinary readers/iterators/clone/tee have byte-level regression coverage. +- Custom lazy/non-native thenables are not a validated Node fetch implementation; + the Node observer does not invoke their overridden `then` to start them. Node's + observer uses V8 promise-parent notifications and native rejection mirrors to + preserve unhandled/handled rejection events and default fatal exits for ordinary + fetch/read promises. Pre-created custom promises consumed before agent startup + are not fully observable. The Node-free edge adapter lacks that Node-specific + rejection-lifecycle mechanism and is not a drop-in process agent. +- An SDK without a processor-removal API can retain one inert agent processor after + shutdown. Agent callbacks stop, application processors remain active, and init does + not add another processor. Optional enable entries own their integration's hook + cleanup when they do not return a disable handle. +- Covered ancestry traversal is bounded to 256 wrapper/native ancestors; esbuild glob + expansion is bounded to 10,000 filesystem entries and 64 recursive directory levels. + +- No outgoing **wire spans** for `node:http`/`node:https`, direct `undici.fetch`/ + `undici.request`, custom fetch implementations captured before preload, WebSocket, + native gRPC, worker processes, browsers or arbitrary non-fetch transports. HTTP/ + Undici diagnostics do harvest credentials independently of span creation. Preload + each separate Node process and worker isolate (globals/registries are not shared + across V8 isolates). OpenAI's included dedicated instrumentation covers its + supported default transport; the global fetch wire hook does not cover node-fetch. +- URL matching is limited to the catalog. Restricted-provider endpoints on a custom + proxy host may not match. No speculative interception of unknown AI protocols. +- Request objects and opaque/pre-encoded multipart streams are never cloned/read; + known provider endpoints retain metadata/body-unavailable capture only. Use native + FormData for text fields and file metadata. Binary request sizes are included only + when the body or declared content length exposes the size. +- A response must be consumed or cancelled to finish capture. Unread or never-ending + responses do not yield completed spans. Force-flush never waits for unread responses. +- Bedrock binary event-stream responses record request/status only. Audio streaming + and base64 variants count bytes but do not reconstruct audio/timestamps. No audio + bytes are exported. +- Registry completeness is not a guarantee that every instrumentor supports every upstream version. Instrumentations depending on obsolete module paths can remain pending. + +## Redistributing the launcher + +The built `dist/agent/` directory is relocatable. Keep the whole directory, including `runtime.cjs`, `hook.mjs`, `hook-register.cjs` and linked `.LEGAL.txt` notices. Use its `preload.cjs` with Node’s `--require` flag, not `--import`; no application installation of `@traceai/auto` is needed for this payload. OpenAI instrumentation is embedded; app-installed instrumentors take precedence. + +The ES5-only preload skips Node below 18 without parsing the modern payload. A runtime without `module.register` has CJS/fetch capture, not ESM hooks. These compatibility guards do not change the supported package engine: Node 20.6+. + +## Legal notices + +The published package includes `dist/THIRD-PARTY-NOTICES`, with the full installed +LICENSE, LICENCE, COPYING and NOTICE files for third-party inputs to all four +launcher/edge bundles. `dist/third-party-manifest.json` maps each bundle to its +third-party package names and versions; first-party `@traceai/*` packages are listed +separately in the notices. These files supplement, and do not replace, the existing +linked `.LEGAL.txt` output. Keep them with any redistributed launcher or edge payload. +The build fails if a bundled third-party package has no readable, nonempty license +text file; a license identifier or linked source comment alone is not sufficient. + +## Maintaining the coverage table + +The README’s library table is generated from `src/integrations.json`. Run `node scripts/generate-readme.cjs` from the package directory after changing the registry; `--check` fails if the checked-in table has drifted. Runtime classes and supported methods remain owned by the individual integrations. diff --git a/typescript/packages/traceai_auto/docs/CONFIGURATION.md b/typescript/packages/traceai_auto/docs/CONFIGURATION.md new file mode 100644 index 000000000..818301396 --- /dev/null +++ b/typescript/packages/traceai_auto/docs/CONFIGURATION.md @@ -0,0 +1,181 @@ +# Configuration reference + +[Back to quick start](../README.md) · [Framework recipes](FRAMEWORKS.md) · [Capture and privacy internals](ARCHITECTURE.md) + +Configure the agent before its first preload or `init()` call. The first configuration is process-wide and terminal, including after failed initialization or shutdown. Restart to change it. `init()` returns the selected tracer provider, or `undefined` if disabled/failed; it does not return a report. + +## Environment and init options + +Unless noted, precedence is **code option → environment → `futureagi.toml` → default**. Code options go directly into `init(options)`; privacy options go under `traceConfig`. Lists in environment variables are comma-separated, case-insensitive names from the [coverage table](../README.md#library-coverage), plus `wire` and `http`. The denylist wins. + +| Environment | `init()` option | Default and behavior | +| --- | --- | --- | +| `FI_API_KEY` | `apiKey` | Unset; resolved as a pair with the secret key | +| `FI_SECRET_KEY` | `secretKey` | Unset; required with API key only for the default FI route | +| `FI_BASE_URL` | `baseUrl` | `https://api.futureagi.com`; base or full collector endpoint. Falls back to the selected project/user profile's `base_url` before the default | +| `FI_PROJECT_NAME` | `projectName` | After code/env/file: `OTEL_SERVICE_NAME` → profile `project` → working-directory basename → `DEFAULT_PROJECT_NAME` | +| `FI_PROJECT_TYPE` | `projectType` | `observe`; `experiment` is opt-in | +| `FI_INSTRUMENTATIONS` | `instrumentations` | Empty list means all eligible integrations; e.g. `openai,wire` | +| `FI_DISABLED_INSTRUMENTATIONS` | `disabledInstrumentations` | Empty list; disable named integrations even if allowlisted | +| `FI_WIRE_CAPTURE` | `wireCapture` | `true`; enables global-fetch capture and transport credential observation when `wire` is also enabled | +| `FI_WIRE_MAX_BODY_BYTES` | `wireMaxBodyBytes` | `4194304` bytes; nonnegative finite values are rounded down, invalid values use the default. A capture cap, not an application payload-size limit | +| `FI_SAMPLE_RATE` | `sampleRate` | `1`; finite values clamped to `[0,1]`, invalid values use `1`. Parent-based root sampling for agent-owned providers only | +| `FI_AUTO_DEBUG` | `debug` | `false`; masked startup JSON report | +| `FI_CONFIG_FILE` | `configFile` | `futureagi.toml`, resolved relative to the working directory | +| `FI_PROFILE` | `profile` | `default`; selects a named profile from the FI project/user files | +| `FI_MASTRA_SPAN_PRODUCER` | `mastraSpanProducer` | `exporter`; parser accepts `exporter` or `bridge`, otherwise warns and uses `exporter`. Does **not** enable Mastra zero-code support in this release | +| `OTEL_SDK_DISABLED` | — | `false`; `true` disables initialization, including when an exporter/provider is supplied | +| — | `exporter` | Unset; standard OTel `SpanExporter`, batched by the agent. No FI credentials required; takes precedence over the default FI exporter | +| — | `spanProcessor` | Unset; explicit OTel `SpanProcessor`. Takes precedence over `exporter` and the default FI route; keeps its selected processor policy, not automatically batched | +| — | `provider` | Existing global provider, otherwise an agent-created provider. Explicit provider selects the instrumentation destination without replacing the global provider | +| — | `traceConfig` | Privacy object described below | + +Except for `hideErrorDetails`, environment booleans are true only when their case-insensitive string value is `true`; `1` is not true. Prefer the literal `true` / `false` forms. + +### Inherited SDK settings + +These are read by fi-core or the underlying OpenTelemetry SDK, not from `futureagi.toml`. The limits and batching defaults below follow the OTel 2.x SDK used by the agent; an application-owned provider/processor may configure its own values. + +| Environment | Default | Scope | +| --- | --- | --- | +| `FI_PROJECT_VERSION_NAME` | `DEFAULT` | fi-core resource version name when the agent constructs a provider; not a project-version option on `init()` | +| `OTEL_SERVICE_NAME` | SDK service default | Application `service.name`; also a fallback for FI project naming | +| `OTEL_RESOURCE_ATTRIBUTES` | Unset | Comma-separated OTel resource attributes; preserved independently of FI project routing | +| `OTEL_BSP_MAX_QUEUE_SIZE` | `2048` spans | Queue bound for the agent's batch processor | +| `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | `512` spans | Export batch size, no larger than the queue bound | +| `OTEL_BSP_SCHEDULE_DELAY` | `5000` ms | Scheduled batch-export interval | +| `OTEL_BSP_EXPORT_TIMEOUT` | `30000` ms | SDK batch-export timeout; auto flush/shutdown still have their shorter five-second deadline | +| `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` / `OTEL_ATTRIBUTE_COUNT_LIMIT` | `128` | Native attribute admission; span-specific configuration takes precedence | +| `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` / `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` | Unlimited | Native string-value limit; observed loss can withhold FI content | +| `OTEL_SPAN_EVENT_COUNT_LIMIT` / `OTEL_SPAN_LINK_COUNT_LIMIT` | `128` each | Native span event/link limits | +| `OTEL_SPAN_ATTRIBUTE_PER_EVENT_COUNT_LIMIT` / `OTEL_SPAN_ATTRIBUTE_PER_LINK_COUNT_LIMIT` | `128` each | Native event/link attribute limits | + +Use `FI_SAMPLE_RATE`, not `OTEL_TRACES_SAMPLER`, for the agent-owned sampler. An existing provider keeps its own sampler. Use `FI_BASE_URL` for the default HTTP export route; gRPC transport, generic OTel exporter selection and exporter headers are not `init()` options. To choose those, supply an exporter or processor configured for your destination. The auto agent explicitly disables fi-core verbose provider/exporter logs; use `FI_AUTO_DEBUG` for its report. + +### Credentials and profiles + +Credentials are resolved **atomically**: both halves must come from the same source. The first complete pair wins: + +1. Explicit `apiKey` + `secretKey` options. +2. `FI_API_KEY` + `FI_SECRET_KEY` environment variables. +3. `api_key` + `secret_key` in the configured `futureagi.toml`. +4. Selected profile in the nearest `.fi/config.toml`, searching upward from the working directory. +5. Selected profile in `$XDG_CONFIG_HOME/fi/config.toml`, or `$HOME/.config/fi/config.toml` (OS home directory if `HOME` is unset). + +An incomplete source is ignored with a value-free warning; its half is never combined with a lower-precedence half. Only credentials, `base_url` and `project` fall back to the profile files. Keep credentials in your environment or secret manager, not a committed config file. + +### Config file + +A credential-free `futureagi.toml` can hold ordinary app settings: + +```toml +project_name = "my-ai-app" +project_type = "observe" +instrumentations = ["openai", "wire"] +wire_capture = true +wire_max_body_bytes = 4194304 +sample_rate = 1 +debug = false +hide_inputs = true +hide_outputs = true +hide_error_details = true +``` + +Keys use snake_case: `base_url`, `project_name`, `project_type`, `instrumentations`, `disabled_instrumentations`, `wire_capture`, `wire_max_body_bytes`, `sample_rate`, `debug`, `mastra_span_producer`, and the privacy keys below. File `debug` maps to `FI_AUTO_DEBUG`. `configFile` and `profile` are selected by code/environment, not by this file. + +This parser supports flat strings (single- or double-quoted), booleans, numbers and single-line string arrays, not full TOML. Comments outside quoted strings are supported. Unknown/unsupported values are ignored. A section header ends flat configuration; only `[profiles.name]` sections in the FI profile files are selected by profile name. Do not put app settings under arbitrary TOML tables. + +## Privacy options + +All code options below are nested under `traceConfig`; file keys are snake_case. Hide controls apply to traceAI-authored values and the agent-owned export view, not an unrelated exporter's native spans. + +| Environment | `traceConfig` option | Default | Effect | +| --- | --- | --- | --- | +| `FI_HIDE_INPUTS` | `hideInputs` | `false` | Hide input values and input-message families | +| `FI_HIDE_OUTPUTS` | `hideOutputs` | `false` | Hide output values and output-message families | +| `FI_HIDE_INPUT_MESSAGES` | `hideInputMessages` | `false` | Hide input message attributes | +| `FI_HIDE_OUTPUT_MESSAGES` | `hideOutputMessages` | `false` | Hide output message attributes | +| `FI_HIDE_INPUT_TEXT` | `hideInputText` | `false` | Hide input text content | +| `FI_HIDE_OUTPUT_TEXT` | `hideOutputText` | `false` | Hide output text content | +| `FI_HIDE_INPUT_IMAGES` | `hideInputImages` | `false` | Hide image input; export masking conservatively hides raw input/message aliases too | +| `FI_HIDE_EMBEDDING_VECTORS` | `hideEmbeddingVectors` | `false` | Hide vectors on library/forwarded/generic-JSON routes; catalog wire embeddings already omit vectors | +| `FI_HIDE_LLM_INVOCATION_PARAMETERS` | `hideLLMInvocationParameters` | `false` | Hide invocation parameters, `gen_ai.request.parameters` and tool schemas in the export view | +| `FI_HIDE_ERROR_DETAILS` | `hideErrorDetails` | Derived | Hide error descriptions, exception messages/stacks and related content; keep error status/type | +| `FI_PII_REDACTION` | `piiRedaction` | `false` | Apply supported PII patterns; not a guarantee of complete PII removal | +| `FI_BASE64_IMAGE_MAX_LENGTH` | `base64ImageMaxLength` | `0` | Value-level image cap. Invalid/negative/fractional values fall back to `0`; positive values do not restore base64 image export | + +When error hiding is unset, any input/output value, message or text hide flag enables it. Invocation-parameter hiding additionally implies error hiding at the export boundary. Image/vector hiding and PII do not imply it. Explicit `true` or `false` overrides the derived error setting; empty/invalid environment values mean derive. Secret and media scrubbing still run with explicit `false`. + +Known-secret redaction is always active on traceAI-owned exports. The registry is bounded to 1024 values of at least eight characters. `registerSecret(value)` adds an exact string without coercing objects. Unknown free-text credentials, transformed secrets and credentials hidden in unparsed bodies may remain unknown; hide sensitive content instead of relying on inference. + +## Bring your own OpenTelemetry destination + +With no complete FI key pair, `init({ exporter })`, `init({ spanProcessor })`, `init({ provider })`, or an existing application provider activates instrumentation **without creating an FI export route**. If neither a destination nor a complete pair exists, instrumentation is disabled. With a complete pair and a provider but no explicit exporter/processor, the agent attempts to attach its FI route to that provider. + +Keep one global provider. Existing OTel JS 1.x/2.x providers retain their sampler and context manager; the agent attaches its processor where supported. An incompatible provider may trigger a warning and an isolated AI provider rather than replacement of the global SDK. + +For an app without an existing SDK, install the exporter you want to use: + +```sh +npm install @opentelemetry/exporter-trace-otlp-http@0.202.0 +``` + +Save as `instrument.mjs`. Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` to your collector's **full traces endpoint** and configure that exporter's authentication separately: + +```js +import { init } from "@traceai/auto"; +import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; + +init({ + projectName: "my-ai-app", + exporter: new OTLPTraceExporter({ + url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, + }), +}); +``` + +Start with your preload instead of the default one: + +```sh +node --import ./instrument.mjs app.mjs +``` + +The agent batches and sanitizes its supplied `exporter` route. A supplied `spanProcessor` is application-owned policy: configure that processor's privacy/export behavior yourself. Agent shutdown invokes the processor/exporter it attaches, but never shuts down an application-owned provider; do not hand it a processor whose lifecycle must remain independent. + +Standard `OTEL_SERVICE_NAME` and `OTEL_RESOURCE_ATTRIBUTES` preserve application identity; FI project routing does not replace `service.name`. The default FI exporter uses `FI_BASE_URL`, not `OTEL_EXPORTER_OTLP_ENDPOINT`. Standard OTel attribute-count/value limits still apply (`OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` / `OTEL_ATTRIBUTE_COUNT_LIMIT`, `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` / `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT`). Native host loss can withhold the FI content view rather than retain a partial value. + +## Lifecycle API + +| Export | Contract | +| --- | --- | +| `init(options?)` | Initialize once; return the same provider on later calls, or `undefined` when disabled/failed | +| `report()` | Masked configuration, `active`, integration statuses and collector state; does not probe the collector | +| `forceFlush()` | Flush completed spans with a five-second deadline; does not consume unread responses | +| `shutdown()` | Idempotent hook removal and bounded drain/shutdown; no supported reinitialization afterwards | +| `registerSecret(value)` | Register an exact sensitive string of at least eight characters; ignore other values | + +## CLI + +The package installs the `traceai-auto` binary. With a local npm installation, invoke it using `npx --no-install traceai-auto` so diagnostics use the package already in your app. + +| Command suffix | Behavior | +| --- | --- | +| `doctor` | Initialize, send one sampled test span, flush, print report and shut down | +| `doctor --no-check` | Initialize and print report, without the test export | +| `bootstrap` | Print detected integrations and suggested `npm install` commands; no installation | +| `bootstrap --install` | Explicitly install missing instrumentors for detected targets, using npm | +| `bootstrap --install --package-manager=pnpm` | Same installation with pnpm; `yarn` is also supported | + +Bootstrap never installs target application libraries. Restart after installing instrumentors. Doctor's exit status can still be zero when the collector fails or the agent is disabled: inspect `active` and `collector` in JSON. Zero sampling or an application-owned destination can legitimately leave the collector `not_tested`. + +| Integration status | Meaning | +| --- | --- | +| `instrumented` | Wire/server hook enabled, or a library patch callback ran | +| `pending` | Hook registered, compatible target import not observed yet | +| `not_installed` | Target library cannot be resolved from this app | +| `missing_instrumentor` | Target found, dedicated integration absent | +| `disabled` | Disabled by config, credentials/destination, or SDK switch | +| `manual` | Explicit wrapper/processor API required | +| `failed` | Load, registration, patch or known version incompatibility; inspect the reason | + +Sources: [`config.ts`](../src/config.ts), [`index.ts`](../src/index.ts), [`cli.ts`](../src/cli.ts), and [fi-core privacy defaults](../../fi-core/src/trace/trace-config/constants.ts). diff --git a/typescript/packages/traceai_auto/docs/FRAMEWORKS.md b/typescript/packages/traceai_auto/docs/FRAMEWORKS.md new file mode 100644 index 000000000..238f95cac --- /dev/null +++ b/typescript/packages/traceai_auto/docs/FRAMEWORKS.md @@ -0,0 +1,159 @@ +# Frameworks and deployment + +[Back to quick start](../README.md) · [Configuration](CONFIGURATION.md) · [Architecture](ARCHITECTURE.md) + +All Node recipes assume `@traceai/auto` is installed in your app and the FI keys/project are configured before startup. Set hide flags before initialization if prompts, responses or error details must not leave the process. + +## ESM and CommonJS + +Prefer the package preload shown in the quick start. It executes before application imports and registers both CJS and supported ESM hooks. A top-level `init()` next to static SDK imports is not equivalent: ESM can load those dependencies before the call runs. Put custom initialization in a separate `--import` file instead. + +The package requires Node **20.6+**. Preload each child process/worker isolate separately. Keep the application working directory consistent so optional target libraries and instrumentors resolve correctly. + +## Next.js + +Use Next's `instrumentation.ts` at the project root (or inside `src` when that is where the app lives). Guard the runtime so the Node agent never enters an edge path: + +```ts +export async function register() { + if (process.env.NEXT_RUNTIME === "nodejs") { + await import("@traceai/auto/register"); + } +} +``` + +For Next versions supporting `serverExternalPackages`, merge these entries into `next.config.mjs`. Keep the SDKs you need to hook external too; this example uses OpenAI: + +```js +export default { + serverExternalPackages: ["@traceai/auto", "openai"], +}; +``` + +Set FI variables on the **server**, never as `NEXT_PUBLIC_*` variables. Trigger a Node-runtime route that calls your model. Do not add this preload to browser code or assume it instruments edge handlers. + +`withTraceAI(config)` from `@traceai/auto/next` is an optional **webpack-only** adapter. It preserves an existing webpack callback and skips client/edge configs, but does not cover Turbopack. Prefer the guarded register hook rather than combining both approaches. + +## Node server bundlers + +These opt-in plugins inject the packaged agent before Node **executable server entries**. They do not instrument browser or edge bundles. Keep AI SDKs external to preserve dedicated library import hooks; global-fetch wire capture can still observe recognized calls when application code is bundled. + +Use only the recipe for your build tool. Each example takes the quick-start `app.mjs` and builds a server executable; install the selected bundler as your normal build dependency. + +### esbuild + +Save as `build.mjs` and run `node build.mjs`: + +```js +import { build } from "esbuild"; +import { traceAI } from "@traceai/auto/esbuild"; + +await build({ + entryPoints: ["app.mjs"], + outfile: "dist/server.mjs", + bundle: true, + platform: "node", + format: "esm", + packages: "external", + plugins: [traceAI()], +}); +``` + +### Vite Node SSR + +Save as `vite.config.mjs`. The plugin applies to production **Node SSR** builds, not development, client builds or `webworker` SSR: + +```js +import { defineConfig } from "vite"; +import { traceAI } from "@traceai/auto/vite"; + +export default defineConfig({ + plugins: [traceAI()], + build: { + ssr: "app.mjs", + outDir: "dist", + rollupOptions: { output: { entryFileNames: "server.mjs" } }, + }, + ssr: { target: "node", external: ["openai"] }, +}); +``` + +### Rollup + +Save as `rollup.config.mjs`. Rollup cannot infer the server platform; `server: true` is required: + +```js +import { traceAI } from "@traceai/auto/rollup"; + +export default { + input: "app.mjs", + external: ["openai"], + output: { file: "dist/server.mjs", format: "es" }, + plugins: [traceAI({ server: true })], +}; +``` + +### webpack 5 + +Save as `webpack.config.mjs`. This recipe emits a CommonJS server executable: + +```js +import { resolve } from "node:path"; +import { TraceAIPlugin } from "@traceai/auto/webpack"; + +export default { + mode: "production", + target: "node", + entry: "./app.mjs", + experiments: { topLevelAwait: true }, + externals: { openai: "commonjs openai" }, + output: { path: resolve("dist"), filename: "server.cjs" }, + plugins: [new TraceAIPlugin()], +}; +``` + +### Deploy the build, not just its entry file + +ESM output includes a small bootstrap at the original entry path and a sibling `*.traceai-app.mjs` payload. Ship **both**, plus generated chunks and sourcemaps. Run the built entry normally; the plugin already injects tracing. + +The injected preload uses an **absolute filesystem path**. Either retain that path in your deployment or pass `agentPath` pointing to the deployment's `preload.cjs` (the file must also exist when the build resolves it). Copy the complete `dist/agent/` directory with its `runtime.cjs`, `hook.mjs`, `hook-register.cjs` and `.LEGAL.txt` files, plus `dist/THIRD-PARTY-NOTICES` and `dist/third-party-manifest.json` alongside the relocated payload. Do not copy only `preload.cjs`. + +| Plugin option | Default | Meaning | +| --- | --- | --- | +| `agentPath` | Installed package's built preload | Absolute or resolvable path to a complete deployed agent payload | +| `server` | Tool-specific detection | Rollup requires `true`; `false` disables injection. Browser/edge detection still prevents injection | + +ESM library/handler entries with exports are rejected by the executable bootstrap; use a runtime preload or a framework register hook instead. `import.meta.url` in the running ESM app identifies the sibling payload, not the bootstrap. Unnamed in-memory esbuild ESM output cannot get a deployable sibling; use an output path or external preload. + +## AWS Lambda and short-lived processes + +Do not rely on process exit in Lambda: the runtime can freeze before a batch exports. Initialize once outside the handler and flush completed spans at each invocation boundary. The fi-core wrapper handles a bounded invocation flush without shutting down a warm provider. + +Install `@traceai/fi-core` directly when importing it in your own code. Save this as a Node ESM handler (for example `index.mjs`); initialize before dynamically importing AI SDKs: + +```js +import { init } from "@traceai/auto"; +import { traceLambdaHandler } from "@traceai/fi-core"; + +const provider = init(); +const { default: OpenAI } = await import("openai"); +const client = new OpenAI(); + +export const handler = traceLambdaHandler(async () => { + const response = await client.chat.completions.create({ + model: process.env.OPENAI_MODEL, + messages: [{ role: "user", content: "Say hello in one word." }], + }); + return { statusCode: 200, body: response.choices[0].message.content }; +}, { provider, maxFlushWaitMillis: 2000, remainingTimeReserveMillis: 250 }); +``` + +Use this wrapper **instead of** an ADOT/upstream Lambda invocation wrapper, not on top of one. The invocation span is metadata-only; child model calls use your configured content policy. Do not call `shutdown()` after each warm invocation. Callback and response-streaming handlers have separate constraints in the [fi-core Lambda reference](../../fi-core/README.md#aws-lambda-invocation-lifecycle). + +In a short-lived ordinary Node program, finish consuming responses, then `await forceFlush()` or `await shutdown()` from `@traceai/auto` before an explicit exit. A five-second bound prevents indefinite shutdown waits but cannot guarantee collector acceptance, cancel a stuck exporter or survive SIGKILL. For an application-owned provider, follow that provider's lifecycle too. + +## Edge and browsers + +Do **not** preload `@traceai/auto/register` in browsers, Cloudflare Workers, Deno isolates or Next edge routes. This package's zero-code agent is Node-only. + +`@traceai/auto/edge` is a low-level, Node-free fetch observer for integration authors. It exports `patchFetch(maxBodyBytes, emit)`, returning a removable hook with pending response-decoding work. It does **not** configure an exporter, read FI keys, apply fi-core `TraceConfig`, propagate OTel context or supply the Node suppression/deduplication policy. It is not an end-to-end edge setup recipe; use a maintained native OTel integration for your runtime rather than copying a custom exporter into your app. diff --git a/typescript/packages/traceai_auto/docs/images/quickstart-trace.png b/typescript/packages/traceai_auto/docs/images/quickstart-trace.png new file mode 100644 index 000000000..1c4c1eb2f Binary files /dev/null and b/typescript/packages/traceai_auto/docs/images/quickstart-trace.png differ diff --git a/typescript/packages/traceai_auto/scripts/generate-readme.cjs b/typescript/packages/traceai_auto/scripts/generate-readme.cjs new file mode 100644 index 000000000..a75fb9928 --- /dev/null +++ b/typescript/packages/traceai_auto/scripts/generate-readme.cjs @@ -0,0 +1,24 @@ +// Keep the public coverage table tied to the registry the loader actually uses. +const { readFileSync, writeFileSync } = require('node:fs'); +const { resolve } = require('node:path'); +const root = resolve(__dirname, '..'); +const integrations = JSON.parse(readFileSync(resolve(root, 'src/integrations.json'), 'utf8')); +const lines = [ + '| Configuration name | Target library | Instrumentation package | Setup |', + '| --- | --- | --- | --- |', + ...integrations.map(entry => { + const mode = entry.enable ? 'Enable entry' : entry.manual ? 'Manual' : 'Import hook'; + const included = entry.name === 'openai' ? ' (included)' : ''; + return `| \`${entry.name}\` | ${entry.targets.map(target => `\`${target}\``).join(', ')} | \`${entry.package}\`${included} | ${mode} |`; + }), +]; +const file = resolve(root, 'README.md'); +const before = readFileSync(file, 'utf8'); +const pattern = /[\s\S]*?/; +if (!pattern.test(before)) throw new Error('README coverage markers are missing'); +const after = before.replace(pattern, `\n${lines.join('\n')}\n`); +if (process.argv.includes('--check')) { + if (before !== after) throw new Error('README coverage is stale; run scripts/generate-readme.cjs'); +} else { + writeFileSync(file, after); +} diff --git a/typescript/packages/traceai_auto/test/docs-consumer.cjs b/typescript/packages/traceai_auto/test/docs-consumer.cjs new file mode 100644 index 000000000..ac450baa1 --- /dev/null +++ b/typescript/packages/traceai_auto/test/docs-consumer.cjs @@ -0,0 +1,138 @@ +// Run only in a disposable installed consumer: this writes app/config files and replaces dist/. +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const http = require('node:http'); +const { spawn } = require('node:child_process'); +const { createRequire } = require('node:module'); +const cwd = path.resolve(process.argv[2]); +const repo = path.resolve(__dirname, '../../../..'); +const pkg = path.join(repo, 'typescript/packages/traceai_auto'); +const read = name => fs.readFileSync(path.join(pkg, name), 'utf8'); +const blocks = text => [...text.matchAll(/```([^\n]*)\n([\s\S]*?)```/g)].map(m => ({lang:m[1],code:m[2]})); +const main = read('README.md'); +const frames = blocks(read('docs/FRAMEWORKS.md')); +const config = blocks(read('docs/CONFIGURATION.md')); +const output = []; +const captured = []; +let base; +const server = http.createServer((req,res) => { + let body=''; req.on('data',b=>body+=b); req.on('end',()=>{ + res.setHeader('content-type','application/json'); + if(req.url==='/tracer/v1/traces') {captured.push(JSON.parse(body));res.end('{}');} + else if(req.url==='/v1/chat/completions') res.end(JSON.stringify({id:'chatcmpl-docs',object:'chat.completion',created:1,model:'example-model',choices:[{index:0,message:{role:'assistant',content:'Hello.'},finish_reason:'stop'}],usage:{prompt_tokens:7,completion_tokens:2,total_tokens:9}})); + else {res.statusCode=404;res.end('{}');} + }); +}); +const spans = () => captured.flatMap(e=>e.resourceSpans||[]).flatMap(r=>r.scopeSpans||[]).flatMap(s=>s.spans||[]); +const cleanEnv = {...process.env}; +for(const key of Object.keys(cleanEnv)) if(key.startsWith('FI_')||key.startsWith('OTEL_')||key==='NODE_OPTIONS') delete cleanEnv[key]; +function environment(extra={}) { return {...cleanEnv,FI_API_KEY:'synthetic-fi-api-123',FI_SECRET_KEY:'synthetic-fi-secret-123',FI_BASE_URL:base,FI_PROJECT_NAME:'my-ai-app',OPENAI_API_KEY:'synthetic-provider-123',OPENAI_BASE_URL:base+'/v1',OPENAI_MODEL:'example-model',...extra}; } +function run(command, extra={}, dir=cwd, timeout=120000) { + return new Promise((resolve,reject)=>{ + const child=spawn('bash',['-c',command],{cwd:dir,env:environment(extra),stdio:['ignore','pipe','pipe']}); + let out='',err=''; child.stdout.on('data',b=>out+=b);child.stderr.on('data',b=>err+=b); + const timer=setTimeout(()=>{child.kill('SIGKILL');reject(new Error(`timeout: ${command}\n${err}`));},timeout); + child.on('error',error=>{clearTimeout(timer);reject(error);});child.on('close',code=>{clearTimeout(timer);code===0?resolve(out):reject(new Error(`exit ${code}: ${command}\n${out}\n${err}`));}); + }); +} +function record(test,extra={}) { const result={consumer:path.basename(cwd),test,status:'PASS',...extra};output.push(result);console.log(JSON.stringify(result)); } +function assertSpans(test,{hidden=false,min=1}={}) { + assert.ok(spans().length>=min,`${test}: ${spans().length} spans`); + const all=JSON.stringify(captured); + assert.ok(!all.includes('synthetic-provider-123')); + if(hidden) {assert.ok(!all.includes('Say hello in one word.'));assert.ok(!all.includes('Hello.'));} + else assert.ok(all.includes('Hello.'),test+' output missing'); + record(test,{spans:spans().length,names:spans().map(s=>s.name)}); +} +function write(name,text) {fs.writeFileSync(path.join(cwd,name),text);} +async function executeSnippets() { + const app=blocks(main).find(b=>b.lang==='js').code; + write('app.mjs',app); + write('app.cjs',app.replace('import OpenAI from "openai";','const OpenAI = require("openai");\n(async () => {')+'\n})();\n'); + const packagePath=path.join(cwd,'package.json'),manifest=JSON.parse(fs.readFileSync(packagePath)); + manifest.scripts={...manifest.scripts,start:'node app.mjs'};fs.writeFileSync(packagePath,JSON.stringify(manifest)); + for(const {code} of blocks(main).filter(b=>b.lang==='sh')) { + if(code.startsWith('npm install')) continue; + captured.length=0; + const command=code.replaceAll('your-api-key','synthetic-fi-api-123').replaceAll('your-secret-key','synthetic-fi-secret-123'); + const result=await run(command); + if(command.includes('bootstrap')) {assert.ok(JSON.parse(result).integrations.length>0);assert.equal(spans().length,0);record('bootstrap command');} + else if(command.includes('doctor')) {const report=JSON.parse(result);assert.equal(report.active,true);if(command.includes('--no-check')) {assert.match(report.collector,/not_tested/);assert.equal(spans().length,0);} else {assert.equal(report.collector,'accepted');assert.ok(spans().some(s=>s.name==='traceai.auto.doctor'));}record(command.trim(),{collector:report.collector});} + else {assert.match(result,/Hello\./);assertSpans(command.trim());} + } + captured.length=0; + const root=fs.readFileSync(path.join(repo, 'README.md'),'utf8').split('## Zero-code tracing for Node.js')[1].split('## Quickstart')[0]; + const rootCommand=blocks(root)[0].code.split('\n').slice(1).join('\n').replaceAll('your-api-key','synthetic-fi-api-123').replaceAll('your-secret-key','synthetic-fi-secret-123'); + await run(rootCommand);assertSpans('root quickstart'); + captured.length=0; + write('instrument.mjs',config.find(b=>b.lang==='js').code); + await run(config.find(b=>b.lang==='sh'&&b.code.startsWith('node ')).code,{FI_API_KEY:'',FI_SECRET_KEY:'',OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:base+'/tracer/v1/traces'}); + assertSpans('custom exporter without FI credentials'); + captured.length=0; + write('futureagi.toml',config.find(b=>b.lang==='toml').code); + await run('node --import @traceai/auto/register app.mjs');assertSpans('documented TOML hides content',{hidden:true}); + fs.unlinkSync(path.join(cwd,'futureagi.toml')); + for(const [name,index,command,entry] of [ + ['esbuild',2,'node build.mjs','dist/server.mjs'], + ['vite',3,'node node_modules/vite/bin/vite.js build','dist/server.mjs'], + ['rollup',4,'node node_modules/rollup/dist/bin/rollup -c','dist/server.mjs'], + ['webpack',5,'node webpack-build.cjs','dist/server.cjs'], + ]) { + captured.length=0;fs.rmSync(path.join(cwd,'dist'),{recursive:true,force:true}); + const filename={esbuild:'build.mjs',vite:'vite.config.mjs',rollup:'rollup.config.mjs',webpack:'webpack.config.mjs'}[name]; + write(filename,frames[index].code); + if(name==='webpack')write('webpack-build.cjs',`const webpack=require('webpack');import('./webpack.config.mjs').then(({default:c})=>webpack(c,(err,stats)=>{if(err||stats.hasErrors()){console.error(err||stats.toString());process.exitCode=1;}}));`); + await run(command);await run(`node ${entry}`);assertSpans('documented '+name+' recipe'); + } + captured.length=0;write('lambda.mjs',frames[6].code); + write('invoke.mjs',`import {handler} from './lambda.mjs';\nfor(let i=0;i<2;i++){const result=await handler({},{awsRequestId:'synthetic-request-'+i,functionName:'readme-example',getRemainingTimeInMillis:()=>5000});if(result.body!=='Hello.')throw new Error('bad response');}\n`); + await run('node invoke.mjs');assertSpans('documented warm Lambda handler',{min:4}); + const req=createRequire(path.join(cwd,'package.json')); + const auto=path.resolve(path.dirname(req.resolve('@traceai/auto')),'../..'); + const {resolveConfig}=req(path.join(auto,'dist/src/config.js')); + const defaults=resolveConfig({configFile:'/missing'},{}); + assert.equal(defaults.sampleRate,1);assert.equal(defaults.wireCapture,true);assert.equal(defaults.wireMaxBodyBytes,4194304);assert.equal(defaults.baseUrl,'https://api.futureagi.com');assert.equal(defaults.projectType,'observe');assert.equal(defaults.debug,false);assert.equal(defaults.mastraSpanProducer,'exporter');assert.equal(defaults.sdkDisabled,false); + assert.deepEqual(defaults.instrumentations,[]);assert.deepEqual(defaults.disabledInstrumentations,[]); + assert.equal(resolveConfig({}, {FI_WIRE_CAPTURE:'1'}).wireCapture,false); + record('configuration defaults against installed package'); + captured.length=0; + const {patchFetch}=req('@traceai/auto/edge');let edge; + const hook=patchFetch(4194304,result=>edge=result); + await (await fetch(base+'/v1/chat/completions',{method:'POST',headers:{'content-type':'application/json'},body:JSON.stringify({model:'example-model',messages:[{role:'user',content:'hello'}]})})).text(); + await Promise.all([...hook.pending]);hook.remove(); + assert.ok(edge);assert.equal(spans().length,0);record('edge observer has no implicit exporter'); +} +async function nextRecipe() { + const dir=path.join(cwd,'next-example');fs.mkdirSync(path.join(dir,'app/api/chat'),{recursive:true}); + fs.writeFileSync(path.join(dir,'instrumentation.ts'),frames[0].code); + fs.writeFileSync(path.join(dir,'next.config.mjs'),frames[1].code); + fs.writeFileSync(path.join(dir,'package.json'),JSON.stringify({name:'readme-next-example',private:true})); + fs.writeFileSync(path.join(dir,'app/layout.js'),`export default function Layout({children}) {return {children};}`); + fs.writeFileSync(path.join(dir,'app/page.js'),`export default function Page(){return
TraceAI documentation example
;}`); + fs.writeFileSync(path.join(dir,'app/api/chat/route.js'),`import OpenAI from 'openai';export const runtime='nodejs';export const dynamic='force-dynamic';export async function GET(){const result=await new OpenAI().chat.completions.create({model:process.env.OPENAI_MODEL,messages:[{role:'user',content:'Say hello in one word.'}]});return Response.json({message:result.choices[0].message.content});}`); + await run('node ../node_modules/next/dist/bin/next build',{NEXT_TELEMETRY_DISABLED:'1'},dir,240000); + captured.length=0; + const child=spawn(process.execPath,['../node_modules/next/dist/bin/next','start','-p','3456'],{cwd:dir,env:environment({NEXT_TELEMETRY_DISABLED:'1'})}); + const closed = new Promise(resolve=>child.once('close',resolve)); + let logs='';child.stdout.on('data',b=>logs+=b);child.stderr.on('data',b=>logs+=b); + try { + let ready=false; + for(let i=0;i<120;i++){try {const r=await fetch('http://127.0.0.1:3456/');if(r.ok){ready=true;break;}}catch{}await new Promise(r=>setTimeout(r,250));} + assert.ok(ready,logs); + const response=await fetch('http://127.0.0.1:3456/api/chat');assert.equal(response.status,200,logs);assert.equal((await response.json()).message,'Hello.'); + for(let i=0;i<80&&!spans().length;i++)await new Promise(r=>setTimeout(r,250)); + assertSpans('documented Next production build/start'); + } finally { + child.kill('SIGTERM'); + const timer=setTimeout(()=>child.kill('SIGKILL'),10000); + await closed; + clearTimeout(timer); + } +} +(async()=>{ + await new Promise(resolve=>server.listen(0,'127.0.0.1',resolve));base=`http://127.0.0.1:${server.address().port}`; + await executeSnippets(); + await nextRecipe(); + if (process.argv[3]) { fs.mkdirSync(process.argv[3],{recursive:true}); fs.writeFileSync(path.join(process.argv[3],path.basename(cwd)+'.json'),JSON.stringify(output,null,2)); } +})().catch(error=>{console.error(error);process.exitCode=1;}).finally(()=>{server.closeAllConnections();server.close();}); diff --git a/typescript/packages/traceai_auto/test/docs.test.ts b/typescript/packages/traceai_auto/test/docs.test.ts new file mode 100644 index 000000000..c0f1f979c --- /dev/null +++ b/typescript/packages/traceai_auto/test/docs.test.ts @@ -0,0 +1,47 @@ +import { readFileSync, existsSync } from "node:fs"; +import { resolve, dirname } from "node:path"; +import { execFileSync } from "node:child_process"; + +const root = resolve(__dirname, ".."); +const repo = resolve(root, "../../.."); +const read = (file: string) => readFileSync(resolve(root, file), "utf8"); + +test("README coverage is generated from the runtime registry", () => { + expect(() => execFileSync(process.execPath, [resolve(root, "scripts/generate-readme.cjs"), "--check"])).not.toThrow(); +}); + +test("configuration documents every init option and privacy control", () => { + const config = read("src/config.ts"); + const reference = read("docs/CONFIGURATION.md"); + const options = config.split("export interface Options {")[1].split("\n}")[0]; + for (const [, option] of options.matchAll(/\b(\w+)\?:/g)) { + expect(reference).toContain(`\`${option}\``); + } + const privacy = config.split('for (const key of [')[1].split(']')[0]; + for (const [, option] of privacy.matchAll(/"(\w+)"/g)) { + expect(reference).toContain(`\`${option}\``); + } +}); + +test("documentation does not require FI credentials for custom exporters", () => { + expect(read("README.md")).not.toContain("It does not bypass the credential requirement"); + expect(read("docs/CONFIGURATION.md")).toContain("No FI credentials required"); +}); + +test("the shared contract does not claim direct Undici wire capture", () => { + const spec = readFileSync(resolve(repo, "spec/auto-instrumentation/README.md"), "utf8"); + expect(spec).not.toContain("covers undici and every fetch-based SDK"); + expect(spec).toContain("not direct `undici.fetch`/`undici.request`"); +}); + +test("package guide links resolve within the repository", () => { + const prefix = "https://github.com/future-agi/traceAI/blob/dev/"; + for (const name of ["README.md", "docs/CONFIGURATION.md", "docs/FRAMEWORKS.md", "docs/ARCHITECTURE.md"]) { + for (const [, url] of read(name).matchAll(/\]\(([^)]+)\)/g)) { + if (url.startsWith("#") || (/^https?:/.test(url) && !url.startsWith(prefix))) continue; + const target = url.split("#")[0]; + const file = target.startsWith(prefix) ? resolve(repo, target.slice(prefix.length)) : resolve(dirname(resolve(root, name)), target); + expect({ url, exists: existsSync(file) }).toEqual({ url, exists: true }); + } + } +});