diff --git a/.oxfmtrc.jsonc b/.oxfmtrc.jsonc index d2be02e64..37e88a61a 100644 --- a/.oxfmtrc.jsonc +++ b/.oxfmtrc.jsonc @@ -9,5 +9,15 @@ // migration-snapshot diff. // lib/llm and lib/thinking-orbs are vendored upstream source in prettier style; // reformatting them would turn every future upstream sync into a whole-file conflict. - "ignorePatterns": [".context", "deploy", "skills", "packages/db/drizzle", "lib/llm", "lib/thinking-orbs"], + // ai-registry/registry.json is a generated artifact copied verbatim from trace-capture + // (see its README); formatting it would make every resync a false diff. + "ignorePatterns": [ + ".context", + "deploy", + "skills", + "packages/db/drizzle", + "lib/llm", + "lib/thinking-orbs", + "packages/domain/src/ai-registry/registry.json", + ], } diff --git a/packages/domain/src/ai-registry/README.md b/packages/domain/src/ai-registry/README.md new file mode 100644 index 000000000..e7e59e1ee --- /dev/null +++ b/packages/domain/src/ai-registry/README.md @@ -0,0 +1,108 @@ +# AI vendor registry (generated artifact) + +`registry.json` is the compiled **per-span AI/agent telemetry classification registry**: the data +that lets Maple look at a single OTLP span and decide _which AI framework emitted it_ and _what +session it belongs to_, without cross-span joins. + +It contains: + +- **`vendors`** — one entry per AI framework, each with **matchers** (every one carrying a unique + integer priority) and **`session_candidates`** describing which attribute keys carry a session/run + identity and how to validate them. Vendors also carry `decoy_keys` / `decoy_values` (keys that look + identifying but are not), `caveats`, and `variants`. +- **`unknown_tier`** — the fallback bucket for spans that are recognisably AI telemetry but match no + known vendor: `unknown:genai`, `unknown:openinference`, `unknown:other`. +- **`algebra`** — the predicate contract the matchers are written in: ops `present`, `eq`, + `key_prefix`, `value_prefix`, plus the attribute-canonicalization rule + (`AnyValue -> String`; bools as `true`/`false`, numbers as decimal strings, array/kvlist as JSON; + duplicate keys resolve first-occurrence-wins among registry-referenced keys). There is deliberately + **no negation, no conjunction, no event access and no JSON traversal** — the algebra is kept small + enough to compile to both SQL and Rust. +- **`session_state_enum`** — the ladder a span lands in (1 = vendor has no session-key rules … 6 = + resolved at session granularity), reduced as `max` over candidates with the hash taken from the + winning candidate, ties broken by candidate order. + +## What is generated and what is not + +`registry.json` is **generated — never hand-edit it.** Edits made here are lost on the next sync and, +worse, silently diverge from the wire-verified seeds that justify every matcher. Treat it as a build +output that happens to be checked in. + +`README.md` and `UPSTREAM.json` are maple-side and hand-maintained — the upstream compiler emits +`registry.json` and nothing else. + +## Provenance + +Generated by `scripts/compile-registry.ts` in the companion repo **trace-capture** +(`https://github.com/MapleTechLabs/trace-capture` — private), compiled from the wire-verified +per-framework seeds at `frameworks//registry-seed.yaml`. + +`UPSTREAM.json` next to this file is the pin: the trace-capture commit that last produced +`registry.json`, the sha256 of the artifact so drift is detectable without a diff, and the date it +was synced. + +"Wire-verified" means each seed was derived from real OTLP captures of that framework, not from +reading its source or docs; the seed format and the review that produces it are specified in +trace-capture's `frameworks/REVIEW_IMPLEMENTATION.md` (§6 is the seed format itself). + +## How to update + +Updates are expected to be **rare** — a new framework, or a vendor changing its wire format. + +1. In the trace-capture repo, edit or add `frameworks//registry-seed.yaml` following + `frameworks/REVIEW_IMPLEMENTATION.md` §6. +2. Run `bun scripts/compile-registry.ts` there to recompile `registry.json`. +3. Keep `bun scripts/verify-seed.ts --all` green there. +4. Copy the regenerated `registry.json` here **verbatim**, and update `UPSTREAM.json` with the new + trace-capture commit, sha256 and `syncedAt`. + +Treat trace-capture as the source of truth; never patch the artifact in maple to fix a +classification bug. + +## Who reads it in maple + +Two implementations, with different jobs: + +- **`apps/ingest` (Rust)** classifies every span at write time and stamps the `AiVendor` / + `AiSessionKey*` columns. This is the only path that classifies live traffic. +- **`packages/domain` (TypeScript)** does _not_ classify on the read path — dashboards and queries + read the stamped columns. It exists for the two jobs Rust cannot do: compiling the same rules to + ClickHouse SQL so a registry fix can re-derive vendors for spans **already on disk** (rollup + rebuilds cannot trust the column being fixed), and backing the differential test that holds the + Rust and SQL evaluators to the same answer span-for-span. + +## Reading the vendor list + +Two things in here regularly surprise people: + +- **There is no `langgraph` vendor — LangGraph spans classify as `langchain`.** LangGraph is + instrumented by LangChain's own tracer, so an individual span carries no evidence separating the + two. Keeping both as vendors would mean two entries matching the same span with one winning + arbitrarily, so the compiler merges them into the wider one; the entry keeps + `renamed_from: "langgraph"`. +- **A single trace can carry spans from several vendors, and usually does.** Classification is + per span, never per trace. `openinference-openai` is the clearest case: that instrumentor carries + the token/model/prompt payload for _any_ framework driving the OpenAI SDK, so in the captures + roughly half the spans of a CrewAI trace classify as `openinference-openai` and the rest as + `crewai`. Anything reading vendors has to aggregate over a trace's spans rather than expect one + label per trace. + +Those are decisions D2 and D3 in the artifact's own `decisions` array. The other two are internal to +the compiler and only matter if you are changing it in trace-capture: **D1** adds a fourth predicate +operator (`value_prefix`, allowed only on scope/span-name pseudo-keys), **D4** assigns priorities in +bands so first-match-wins is deterministic. + +## Validation + +The **capture corpus is deliberately not vendored.** trace-capture holds the OTLP captures every +seed was derived from; they are large, contain raw third-party payloads, and are only useful next to +the replay tooling that reads them. + +Consequently: + +- **Corpus-replay verification against seed goldens is an on-demand local gate**, run from the + trace-capture repo — not from maple, and not in CI. +- **Maple's CI relies on synthetic differential and property tests** over this artifact: the TS and + Rust evaluators must agree span-for-span, and the algebra's invariants + (unique priorities, band ordering, `value_prefix` pseudo-key restriction, session-state reduction) + are asserted directly against `registry.json`. diff --git a/packages/domain/src/ai-registry/UPSTREAM.json b/packages/domain/src/ai-registry/UPSTREAM.json new file mode 100644 index 000000000..2d16212af --- /dev/null +++ b/packages/domain/src/ai-registry/UPSTREAM.json @@ -0,0 +1,10 @@ +{ + "repo": "https://github.com/MapleTechLabs/trace-capture", + "ref": "main", + "sha": "cb29d8434a5919fd106278947ed3e262456407e5", + "generatedBy": "bun scripts/compile-registry.ts", + "compiledFrom": "frameworks//registry-seed.yaml (20 seeds + 1 synthesized vendor)", + "artifact": "registry.json", + "sha256": "226031079a9b273a3f799998ad8152c727a62f1c0a45593d1b884f52e2718595", + "syncedAt": "2026-08-11" +} diff --git a/packages/domain/src/ai-registry/registry.json b/packages/domain/src/ai-registry/registry.json new file mode 100644 index 000000000..4e0b62e59 --- /dev/null +++ b/packages/domain/src/ai-registry/registry.json @@ -0,0 +1,3515 @@ +{ + "registry_version": 1, + "generated_by": "scripts/compile-registry.ts", + "compiled_from": "20 seeds + 1 synthesized vendor(s)", + "decisions": [ + "D1 algebra: +value_prefix(pseudo_key, prefix) [pseudo-keys only]; negation/conjunction/events/JSON excluded", + "D2 rename: langgraph -> langchain (dialect not span-locally separable)", + "D3 added vendor: openinference-openai (shared instrumentor scope, unclaimed by design)", + "D4 priorities: sufficient(3xxxx) > vendor-attr(2xxxx) > unknown-tier(1xxxx), mechanical within bands" + ], + "algebra": { + "ops": [ + "present", + "eq", + "key_prefix", + "value_prefix" + ], + "value_prefix_pseudo_keys": [ + "scope.name", + "span.name", + "scope.version", + "scope.schema_url" + ], + "canonicalization": "AnyValue->String (bool: true/false, numbers: decimal string, array/kvlist: JSON); duplicate keys first-occurrence-wins among registry-referenced keys" + }, + "session_state_enum": { + "1": "vendor has no session-key rules", + "2": "span not session-authoritative", + "3": "authoritative, key absent", + "4": "key present, failed validation", + "5": "resolved at run/instance/user granularity", + "6": "resolved at session granularity", + "reduction": "max over candidates; hash from winning candidate, ties by candidate order" + }, + "unknown_tier": [ + { + "bucket": "unknown:genai", + "predicate": { + "op": "present", + "key": "gen_ai.operation.name" + }, + "priority": 19999 + }, + { + "bucket": "unknown:openinference", + "predicate": { + "op": "present", + "key": "openinference.span.kind" + }, + "priority": 19998 + }, + { + "bucket": "unknown:other", + "predicate": { + "op": "key_prefix", + "prefix": "llm." + }, + "priority": 19997 + }, + { + "bucket": "unknown:other", + "predicate": { + "op": "key_prefix", + "prefix": "traceloop." + }, + "priority": 19996 + }, + { + "bucket": "unknown:other", + "predicate": { + "op": "key_prefix", + "prefix": "ai." + }, + "priority": 19995 + } + ], + "vendors": [ + { + "vendor": "agno", + "seed": "frameworks/agno/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.agno" + }, + "owned_by": "library", + "source": "wire", + "justification": "The scope name is the instrumentor's own module path, passed as get_tracer(__name__, __version__, tracer_provider) in openinference/instrumentation/agno/__init__.py — library-owned, namespaced, per-package (each openinference-instrumentation-* artifact gets its own scope, so there is no cross-vendor collision inside the OpenInference family). It catches 17/17 + 20/20 spans, including the 11 + 15 LLM/TOOL spans that carry no agno.* attribute at all and the 3 continue_run orphans. The instrumentor only wraps agno agent/team/workflow/tool/model entry points, so no non-AI span can land in this scope.", + "priority": 39999 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "agno." + }, + "source": "wire", + "priority": 29999 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "any_of": [ + { + "op": "eq", + "key": "openinference.span.kind", + "value": "AGENT" + }, + { + "op": "key_prefix", + "prefix": "agno.workflow." + } + ] + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "A" + }, + { + "key": "agno.run.id", + "authority_predicate": { + "any_of": [ + { + "op": "eq", + "key": "openinference.span.kind", + "value": "AGENT" + }, + { + "op": "key_prefix", + "prefix": "agno.workflow." + } + ] + }, + "validation": [ + "non_empty" + ], + "granularity": "run", + "verdict": "A" + } + ], + "decoy_keys": [ + { + "key": "agno.agent.id", + "source": "wire", + "why": "The Agent object's id (agno/_runs_wrapper.py:168). CONSTANT across all 6 traces of agno_user (\"user-assistant\") — the single most convincing false session key in this framework: root-only, stable across turns, stable across the whole process, and adjacent to the real key in the same namespace. It is agent identity: two concurrent users of the same Agent share it." + }, + { + "key": "agno.team.id", + "source": "wire", + "why": "Team analogue of agno.agent.id (\"research-orchestrator\", _runs_wrapper.py:145). Same trap, and it is the key on the agno_agents ROOT span, so a root-only reader is maximally likely to grab it." + }, + { + "key": "graph.node.id", + "source": "wire", + "why": "Looks like a durable node identity but is regenerated per execution (_generate_node_id() at every start_span): 6 distinct values for the SAME agent across agno_user's 6 turns. Per-run, not per-node and not per-session. graph.node.parent_id points at another such ephemeral id." + }, + { + "key": "service.instance.id", + "source": "wire", + "why": "OTel-SDK-minted uuid4, one per PROCESS. Constant within a capture and session-shaped, but it is process identity — in a long-lived server it is constant across every session forever." + }, + { + "key": "llm.input_messages.N.message.tool_calls.M.tool_call.id", + "source": "wire", + "why": "Provider tool-call ids echoed into the message history; they repeat across the HITL request/resume span pair, which makes them look like a correlation key. They correlate one tool call, not a session." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "session_id_default_is_instance_granularity", + "text": "The corpus' sharpest session trap. agno/agent/_session.py:52-58 mints a uuid4 when no session_id is passed AND assigns it back to the Agent instance ('sticky'), so session.id is ALWAYS present (verdict A) but in a long-lived server sharing one Agent object it is a single constant for every user and every conversation, indistinguishable on the wire from a well-behaved per-conversation id. No validation token can detect this — it is well-formed, non-empty and not a sentinel. NOTES.md previously claimed the key appears only when the app passes session_id=; the source says otherwise and this seed follows the source." + }, + { + "id": "continue_run_is_uninstrumented", + "text": "openinference-instrumentation-agno 1.0.1 wraps the four _run entry points and nothing on the resume path, so agent.continue_run() — agno's OWN native HITL completion call — emits no AGENT span. Its LLM and tool spans surface as parentless roots in fresh traces with neither session.id nor agno.run.id. In agno_user this is 3 spans / 3 traces (33% of the traces, 100% of the approval-execution evidence). The denied turn's rejection and the approved turn's actual delete_file execution are both stranded. This is a fixture of a real upstream gap, not a harness artefact." + }, + { + "id": "context_leak_merges_top_level_runs", + "text": "The instrumentor uses start_span + use_span(end_on_exit=False) and does not detach cleanly on the async path, so a subsequent top-level run inherits the previous run's context. In agno_agents summary_agent.arun (a completely separate await summarizer.arun call) is parented to research_orchestrator.arun and the whole process lands in ONE trace — with the child STARTING 3 ms AFTER its parent ENDED (parent +0..12245 ms, child +12248..15391 ms). Consumers must tolerate children whose time range falls outside the parent's, parent durations that do not bound their subtree, and unrelated sibling runs silently merged. This also produces the root_middle arrival: the root ends before its last children." + }, + { + "id": "root_middle_arrival", + "text": "agno_agents is the corpus' root-middle case: the root research_orchestrator.arun arrives in record seq 2 of 4 because it ended before summary_agent.arun (which the context leak reparented under it) even started. A root-anchored ingest path would have to buffer past the root's arrival, not just until it — the assumption 'the root is the last span of its trace' fails here in the opposite direction from BatchSpanProcessor's usual root-last ordering." + }, + { + "id": "token_count_total_only_on_ainvoke", + "text": "llm.token_count.total is set ONLY in the ainvoke wrapper (_model_wrapper.py:519-520); the invoke, invoke_stream and ainvoke_stream wrappers omit it while all four set prompt/completion. Wire-confirmed: 0/9 LLM spans in agno_user (sync run()) vs 9/9 in agno_agents (async arun()). Any dashboard reading llm.token_count.total silently reports zero total tokens for every synchronous agno app; prompt+completion must be summed instead." + }, + { + "id": "span_names_do_not_identify_the_model", + "text": "LLM span names are the agno CLASS (OpenRouter.invoke / OpenRouter.ainvoke) for every model — agno_agents mixes openai/gpt-4o-mini (6) and anthropic/claude-haiku-4.5 (3) under one span name. Only llm.model_name carries the truth, and llm.provider is the agno class name 'OpenRouter', not a semconv provider value. Reinforces that span-name rules are useless here (and this seed writes none)." + }, + { + "id": "zero_gen_ai_by_default", + "text": "Under default config agno emits NOT ONE gen_ai.* attribute across all 37 spans — it is a pure OpenInference-dialect framework. Any ingest path that keys AI-ness on gen_ai.operation.name sees zero agno spans. Only the openinference-genai-semconv variant changes this, and it is off by default." + }, + { + "id": "all_spans_internal_no_events", + "text": "37/37 spans are kind=1 (INTERNAL); no SERVER/CLIENT spans, no span events, no logs, no metrics. The one ERROR span (fetch_transport_data, code=2 with message 'transport data service unavailable (503)') is native agno: the tool raises, FunctionCall records the failure, the parent AGENT span stays OK and the run degrades gracefully. Span kind carries zero classification signal." + }, + { + "id": "attribute_typing_is_mixed", + "text": "Unlike the Java frameworks, values keep their OTLP types: token counts arrive as intValue, llm.tools.N.tool.json_schema / llm.invocation_parameters / tool.parameters / agno.tools as JSON stringValue. Canonical stringification (the verifier's rule) is what makes eq() comparable across them." + } + ], + "variants": [ + { + "name": "openinference-genai-semconv", + "trigger": "OPENINFERENCE_ENABLE_GENAI_SEMCONV=true (or TraceConfig( enable_genai_semconv=True)); default false — openinference/instrumentation/ config.py:109,118,259-263", + "effects": "_spans.py:74-80 dual-writes OTel GenAI attributes from the OpenInference ones in OpenInferenceSpan.end(): gen_ai.operation.name, gen_ai.provider.name, gen_ai.agent.name, gen_ai.request.model, gen_ai.usage.*, gen_ai.input_messages / output_messages, gen_ai.tool.*, and — critically — gen_ai.conversation.id derived verbatim from session.id (_genai_conversion.py:67-69). Under this variant the framework stops being zero-gen_ai.*: a generic gen_ai.operation.name matcher starts firing on every agno span, co_occurring_families gains otel_gen_ai_semconv, and a SECOND session candidate (gen_ai.conversation.id) appears on exactly the same spans that already carry session.id. It does NOT become root-start-readable: the dual-write happens in end(), after the sampler has run.", + "exercised_in_captures": false + }, + { + "name": "using-session-context-manager", + "trigger": "App wraps calls in openinference.instrumentation.using_session(id) / using_attributes(session_id=id) — context_attributes.py:98-112, read by get_attributes_from_context() at every wrapper's start_span", + "effects": "session.id stops being AGENT-only: _model_wrapper.py:370,427,498,561 and _tools_wrapper.py:101,161 splice the same context attributes into LLM and TOOL spans, so carried_by grows from 6/17 to 17/17. This is the ONLY mechanism that puts a session key on the continue_run orphan spans (they are ordinary FunctionCall/model spans), i.e. the only customer-side fix for unlinkable_populations. Candidate authority_predicate would need the CHAIN/LLM/ TOOL kinds added; left out of the shipped rule because the default deployment does not do this.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "claude_agent_sdk", + "seed": "frameworks/claude_agent_sdk/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "resource", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "service.name", + "value": "claude-code" + }, + "source": "source_code", + "justification": "NATIVE DEFAULT, not observed on the wire here. Fc7() seeds the resource with ATTR_SERVICE_NAME=\"claude-code\" and ATTR_SERVICE_VERSION=, then merges osDetector/hostDetector/envDetector on top — so OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES wins, which is exactly what the harness did (service.name=claude-agent-sdk- [H]). Kept sufficient: false because the value is a plain, squattable service name that any app may set, and because it is absent whenever the customer names their service. service.version alone is unusable — it is a bare semver.", + "priority": 29998 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "com.anthropic.claude_code.tracing" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded in the CLI bundle: HV() = trace.getTracer( \"com.anthropic.claude_code.tracing\", \"1.0.0\"). Reverse-DNS namespaced, library-owned, and the ONLY tracer the CLI ever creates — 23/23 + 32/32 spans land here and no non-CLI span can. Its version is the DIALECT version (1.0.0), not the CLI version, so it is stable across CLI releases. This is the primary discriminator; the attr_matchers below exist only for the scope-rewritten case.", + "priority": 39998 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "com.anthropic.claude_code.events" + }, + "owned_by": "library", + "source": "wire", + "justification": "The LOG scope (getLogger(\"com.anthropic.claude_code.events\", )). Never carries spans, so it contributes nothing to the span goldens; recorded because maple's log ingest needs the same vendor mapping and because the scope VERSION here tracks the CLI (2.1.154), unlike the trace scope.", + "priority": 39997 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "com.anthropic.claude_code" + }, + "owned_by": "library", + "source": "wire", + "justification": "The METRIC scope (getMeter(\"com.anthropic.claude_code\", )). Never carries spans. Recorded for the same reason as the events scope; note it is a strict string PREFIX of the other two, which is why the three must be written as three exact eq() rules (see algebra_violations).", + "priority": 39996 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.type", + "value": "interaction" + }, + "source": "wire", + "priority": 29997 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.type", + "value": "llm_request" + }, + "source": "wire", + "priority": 29996 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.type", + "value": "tool" + }, + "source": "wire", + "priority": 29995 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.type", + "value": "tool.execution" + }, + "source": "wire", + "priority": 29994 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.type", + "value": "tool.blocked_on_user" + }, + "source": "wire", + "priority": 29993 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.type", + "value": "hook" + }, + "source": "source_code", + "priority": 29992 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.type", + "value": "subagent.spawn" + }, + "source": "source_code", + "priority": 29991 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "op": "present", + "key": "span.type" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "A" + }, + { + "key": "user.id", + "authority_predicate": { + "op": "present", + "key": "span.type" + }, + "validation": [ + "non_empty" + ], + "granularity": "user", + "verdict": "A" + } + ], + "decoy_keys": [ + { + "key": "prompt.id", + "source": "wire", + "why": "Log records only (20/20 and 29/29). A bare uuid4 that is stable for the whole of one user turn — session-shaped and adjacent to session.id in the same attribute set, but claude_agent_sdk_user shows 6 distinct values under one session. Joining on it fragments a conversation into turns." + }, + { + "key": "agent_id", + "source": "wire", + "why": "Opaque 17-hex subagent instance id on 7/32 llm_request and 3/32 tool spans in claude_agent_sdk_agents (4 distinct values, one per worker). Identifies a subagent invocation inside one session; the human-readable role lives on the PARENT tool span as subagent_type. Never a session." + }, + { + "key": "user.account_uuid", + "source": "wire", + "why": "Wire-observed in captures/claude_agent_sdk_probe/_probe2 (first-party auth), on every span. A bare UUID sitting next to session.id in the same base attribute set — maximally session-shaped, and actually the ACCOUNT identity: constant for the life of the account. The same applies to its companions organization.id, user.account_id and user.email (the last is PII and must not be indexed)." + }, + { + "key": "tool_use_id", + "source": "wire", + "why": "Log records only (5/20 and 14/29). Anthropic toolu_* id; correlates a tool_decision to its tool_result, not a session." + }, + { + "key": "client_request_id", + "source": "wire", + "why": "Wire-observed on llm_request in the probe captures (absent through an LLM gateway). Per-HTTP-attempt id — the finest granularity in the dialect." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [ + { + "key": "capture.id" + }, + { + "key": "deployment.environment" + }, + { + "key": "service.name" + } + ], + "caveats": [ + { + "id": "telemetry_is_the_cli_not_the_sdk", + "text": "The Claude Agent SDK emits nothing. query() spawns the Claude Code CLI and the CLI is the OTel producer, so 100% of the configuration surface is environment variables passed through options.env — and in TypeScript options.env REPLACES the child environment rather than merging it (Python merges), so a customer who sets env without spreading process.env silently loses PATH, auth and every OTEL_* var. Version skew is a live risk: the SDK ships a bundled CLI (2.1.141 here) that differs from the machine's (2.1.154), and the dialect version travels in the scope, not the package." + }, + { + "id": "trace_per_turn_not_per_session", + "text": "There is NO session-spanning root span. Six turns produced six traces sharing one session.id; a subagent fan-out stays inside its turn's trace. Any product view that equates 'trace' with 'conversation' is wrong for this vendor — the conversation is the session.id group-by, and that group-by is the ONLY thing holding the turns together." + }, + { + "id": "root_arrives_last_every_time", + "text": "4/4 multi-batch traces are root-last, and the claude_agent_sdk_agents trace arrived across EIGHT OTLP records with its claude_code.interaction root in record 8 of 8 (seq 24) — 19.4 s after the trace began. This is BatchSpanProcessor semantics (spans queue on end, parents end last), not a quirk. Any per-trace decision keyed on the root is structurally impossible. It costs nothing here because every span carries both the classifier and the session key, so all 7 traces are classified AND session-keyed in their first record." + }, + { + "id": "approval_outcome_is_span_invisible", + "text": "claude_code.tool.blocked_on_user reports decision=unknown, source=unknown for every APPROVED call (7/7 in agents, 2/3 in user); only the DENIED call shows decision=reject. The true outcome is in the tool_decision LOG record (accept/user_temporary vs reject/user_reject). The span-only signal for approval is structural: an approved call has a claude_code.tool.execution child, a denied one has none. HITL is therefore not expressible in this seed's predicate algebra at all." + }, + { + "id": "numbers_are_strings", + "text": "Almost every numeric and boolean value arrives as OTLP stringValue — input_tokens=\"1228\", success=\"true\", duration_ms=\"3356\", attempt=\"1\" — and gen_ai.response.finish_reasons arrives as the JSON-ish string [\"end_turn\"], not an arrayValue. Duration is duplicated and stringly typed: duration_ms on tool/ llm_request/tool.execution/tool.blocked_on_user, interaction.duration_ms on the root (the root does NOT carry duration_ms), alongside the real start/end timestamps." + }, + { + "id": "provider_attribution_is_wrong", + "text": "gen_ai.system is the hardcoded literal \"anthropic\" on every llm_request (source zv7), regardless of where the request actually went — every call in these captures was served by OpenRouter via ANTHROPIC_BASE_URL. gen_ai.request.model / model carry the truth (anthropic/claude-haiku-4.5, anthropic/claude-sonnet-4.5). Provider attribution keyed on gen_ai.system is silently wrong for any LLM-gateway deployment, which is the normal enterprise setup for this CLI." + }, + { + "id": "mcp_tool_name_placeholder_in_logs", + "text": "tool_decision and tool_result log records report tool_name=\"mcp_tool\" for EVERY MCP tool (source B7(): any name starting with mcp__ collapses to the literal mcp_tool), and mcp_server.name / mcp_tool.name are the literal \"custom\" unless the server is on an internal allowlist. The real identity is only in tool_parameters ({\"mcp_server_name\":\"tracetools\",\"mcp_tool_name\":\"get_weather\"}). SPANS are unaffected — claude_code.tool carries the fully-qualified mcp__tracetools__get_weather. Log-derived tool metrics will collapse every MCP tool into one bucket." + }, + { + "id": "no_telemetry_sdk_resource_attrs", + "text": "The resource is built from scratch by Fc7() (resourceFromAttributes + osDetector + host.arch + envDetector), never from defaultResource(), so telemetry.sdk.name / .language / .version are ABSENT — 55/55 spans, 49/49 log records and every metric block carry exactly {service.name, service.version, os.type, os.version, host.arch} plus whatever OTEL_RESOURCE_ATTRIBUTES adds. No SDK-language signal is available to ingest, and service.version (2.1.154) is the CLI version, which is also the logs/metrics scope version — a usable cross-check." + }, + { + "id": "subagent_nesting_is_real", + "text": "Subagent spans hang off the orchestrator's claude_code.tool.execution for that Agent/Task call, so a 4-worker delegation is ONE 32-span trace with real parent links, and the three parallel workers genuinely overlap on the wire (weather 2741→ 5393 ms, transport 3350→6372 ms, budget 3066→8586 ms from interaction start). The SDK option is named `Task` but the tool surfaces in spans and messages as `Agent`; the readable role is subagent_type on the parent claude_code.tool span (4/7), while the child spans carry only the opaque agent_id." + }, + { + "id": "error_representation", + "text": "The only ERROR-status span in the corpus is the native claude_code.tool.execution for fetch_transport_data: status {code: 2, message: \"transport data service unavailable (503)\"} plus success=\"false\" and an `error` attribute — set by the CLI itself, with no harness involvement. There are ZERO exception span events; the only span event in the dialect is gen_ai.request.attempt. Failed LLM calls instead produce llm_request spans with success=false/error/status_code plus api_error LOG records (wire-observed in the probe captures)." + }, + { + "id": "pruned_agents_capture", + "text": "claude_agent_sdk_agents was pruned after the review's first pass: a superseded failed run (28 spans, llm_request 404 error spans from OpenRouter dropping anthropic/claude-3.5-haiku) was removed, leaving the 25 records / 32 spans / 1 trace recorded in `captures:` above. NOTES.md's older 41-record / 60-span / 2-trace description is the pre-prune superset and has been corrected." + }, + { + "id": "probe_captures_excluded", + "text": "captures/claude_agent_sdk_probe and _probe2 (3 records each) are deliberately NOT in `captures:` and not in the goldens. They are cited as source: wire evidence for the firstparty-identity variant (organization.id / user.email / user.account_uuid / user.account_id / client_request_id on every span) and for the api_error log event, neither of which the golden captures exercise." + } + ], + "variants": [ + { + "name": "traces-off", + "trigger": "CLAUDE_CODE_ENHANCED_TELEMETRY_BETA (or ENABLE_ENHANCED_TELEMETRY_BETA) unset — THE DEFAULT. Source Xv8()/Kz3(): the TracerProvider is only built when CLAUDE_CODE_ENABLE_TELEMETRY is truthy AND this beta flag is truthy.", + "effects": "ZERO spans. Logs and metrics still export normally, so a default-configured Claude Code customer is a logs+metrics-only tenant: no traces section applies, the session key survives only on log records and metric points, and every §4 classification rule in this file that targets spans is dead. This is the single biggest determinant of what maple receives from this vendor.", + "exercised_in_captures": false + }, + { + "name": "session-id-off", + "trigger": "OTEL_METRICS_INCLUDE_SESSION_ID=false (default true — source Hh5/S56()). Despite the METRICS in the name it gates the shared base-attribute builder KNH(), which feeds SPANS, LOG RECORDS and METRIC POINTS alike.", + "effects": "session.id disappears from every span, every log record and every metric point. The session candidate drops to state 3 everywhere and all traces become unsessioned; user.id (never gated) is the only remaining identity key.", + "exercised_in_captures": false + }, + { + "name": "firstparty-identity", + "trigger": "CLI authenticated to a first-party Claude account (OAuth/API key) instead of an LLM-gateway base URL; plus OTEL_METRICS_INCLUDE_ACCOUNT_UUID (default true) and OTEL_METRICS_INCLUDE_VERSION / OTEL_METRICS_INCLUDE_ENTRYPOINT (default false).", + "effects": "KNH() adds organization.id, user.email, user.account_uuid, user.account_id (and optionally app.version, app.entrypoint) to EVERY span, log record and metric point; llm_request additionally gains client_request_id, request_id and gen_ai.response.id from the Anthropic response headers. Adds four new session-looking decoy keys and a PII key (user.email). Wire-observed in captures/claude_agent_sdk_probe and _probe2, which are excluded from the goldens.", + "exercised_in_captures": false + }, + { + "name": "beta-tracing-detailed", + "trigger": "ENABLE_BETA_TRACING_DETAILED=1 AND BETA_TRACING_ENDPOINT= (source K2()). Also requires an internal gate, so it is not generally reachable.", + "effects": "Adds the claude_code.hook span (span.type=hook, hook_event/hook_name/ num_hooks/num_success/num_blocking/...), adds new_context / system_prompt_hash / system_prompt_preview / system_prompt_length attributes to interaction and llm_request, adds a system_prompt log event, and mirrors traces+logs to a SECOND OTLP endpoint. claude_code.subagent.spawn is gated by d5H(), which returns a hardcoded false in 2.1.154 — that span type cannot be emitted by this build at all.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "crewai", + "seed": "frameworks/crewai/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.crewai" + }, + "owned_by": "library", + "source": "wire", + "justification": "The scope name is the instrumentor package's own module __name__ passed to trace_api.get_tracer(__name__, __version__, tracer_provider) (openinference/instrumentation/crewai/__init__.py:120-121); version is the package's __version__. Library-owned, namespaced, stable, and independent of anything the customer names. Catches 12/12 and 9/9 CrewAI spans in the captures — CHAIN kickoff, AGENT _execute_core and TOOL run — including the TOOL spans that carry no crew_* key at all. No non-CrewAI span was observed in this scope.", + "priority": 39995 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "crewai.telemetry" + }, + "owned_by": "library", + "source": "source_code", + "justification": "CrewAI's OWN product-analytics tracer name, hardcoded at 21 call sites in crewai/telemetry/telemetry.py (trace.get_tracer(\"crewai.telemetry\")). Not observed in these captures because the fixture sets CREWAI_DISABLE_TELEMETRY=true, but it reaches a CUSTOMER's collector whenever the customer left telemetry at its default and installed a TracerProvider before building a Crew (see variants.crewai-product-telemetry-adopts-app-provider). Vendor is unambiguously crewai; the spans are product analytics, not AI operations, so a consumer must not treat them as agent/LLM activity.", + "priority": 39994 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "crew_" + }, + "source": "wire", + "priority": 29990 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "present", + "key": "task_key" + }, + "source": "wire", + "priority": 29989 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "present", + "key": "tool.result_as_answer" + }, + "source": "wire", + "priority": 29988 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "present", + "key": "tool.description_updated" + }, + "source": "wire", + "priority": 29987 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "present", + "key": "tool.cache_function" + }, + "source": "wire", + "priority": 29986 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "flow_" + }, + "source": "source_code", + "priority": 29985 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "flow.node." + }, + "source": "source_code", + "priority": 29984 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.crewai" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "C" + } + ], + "decoy_keys": [ + { + "key": "crew_key", + "source": "wire", + "why": "An md5 of the crew's CONFIGURATION, not an identity: 7/25 and 6/17 spans, a single value per capture. crew.py:879-883 computes it as an md5 over the pipe-joined agent.key list plus task.key list — pure configuration text, no instance id — so two different customers running the same published crew template produce the SAME crew_key forever, and every run of one deployment shares it. Joining on it would merge unrelated tenants' conversations." + }, + { + "key": "crew_id", + "source": "wire", + "why": "uuid4 minted per Crew OBJECT (str(crew.id), _wrappers.py:482). It is constant across all 6 turns here only because the fixture builds one Crew and calls kickoff() once; a server that constructs a Crew per request gets a fresh value per RUN, and a module-level Crew gets one value for the whole PROCESS lifetime. Instance granularity masquerading as session granularity — the exact trap §3 warns about." + }, + { + "key": "task_key", + "source": "wire", + "why": "md5 of the task's description + expected_output (task.py:596-601) (6/25, 5/17, one distinct value per task). Stable across runs and across customers using the same task text; identifies a task TEMPLATE, not an execution and not a session." + }, + { + "key": "task_id", + "source": "wire", + "why": "uuid4 per Task object (6/25, 5/17). Sub-run granularity — six distinct values inside one fixture 'conversation'." + }, + { + "key": "graph.node.id", + "source": "wire", + "why": "The agent ROLE string (\"Helpful Assistant\", \"weather_worker\"). Constant across every turn in crewai_user, so it looks like a stable correlation key; it identifies an agent definition, not a session or user." + }, + { + "key": "coding_agent", + "source": "source_code", + "why": "Set process-wide by CrewAI's CommonAttributesSpanProcessor when product telemetry is enabled (telemetry.py:112-121, utils.detect_coding_agent). Values are a closed set (claude_code | cursor | codex | vscode_terminal | non_interactive | unknown) — an environment fingerprint of the machine, never an identity. Not observed here (telemetry disabled)." + }, + { + "key": "service.name", + "source": "wire", + "why": "Harness-set here [H] (crewai-user-flow / crewai-orchestration); in a real deployment it is the app's service name — process identity, never session identity." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "two_mandatory_instrumentors", + "text": "A useful CrewAI trace requires TWO OpenInference packages, and only one of them is crewai's. Installing openinference-instrumentation-crewai alone yields CHAIN/AGENT/TOOL spans with zero token counts, zero model ids and zero prompts — 52% and 47% of the spans in these captures (the entire LLM layer) come from openinference-instrumentation-openai. CrewAI 1.x demoted LiteLLM to an optional extra and calls providers through the native openai SDK, so openinference-instrumentation-litellm would capture nothing; the correct pairing is version-dependent and will change again if CrewAI switches client." + }, + { + "id": "crewai_product_telemetry_can_leak_into_customer_collectors", + "text": "CrewAI ships anonymous product telemetry that is ON BY DEFAULT and installed at import time, exporting to a hardcoded https://telemetry.crewai.com:4319 (telemetry/constants.py). Two consequences for maple. (a) If the customer already installed a real TracerProvider, Telemetry.set_tracer() (telemetry.py:254-268) does NOT create its own — CrewAI's `Crew Created` / `Crew Execution` / `Task Execution` / `Tool Usage` / `Human Feedback` spans are then created by the CUSTOMER's tracer under scope `crewai.telemetry` and shipped to the customer's backend. This is the ordinary OpenInference setup order, so it is likely, not exotic. (b) Regardless of which provider wins, CommonAttributesSpanProcessor is attached to the customer's provider and stamps `coding_agent` on every span it emits. The only clean opt-out is CREWAI_DISABLE_TELEMETRY=true set BEFORE crewai is imported (what this fixture does); OTEL_SDK_DISABLED=true also works but kills the customer's own spans too." + }, + { + "id": "span_names_are_customer_data", + "text": "Every CrewAI span name is built from user-supplied strings: `>.kickoff`, `.._execute_core`, `.run`, `.`. Unnamed crews/flows fall back to a raw uuid in the name (_wrappers.py:249, 267), making span name a high-cardinality, PII-adjacent field. Combined with the algebra's lack of a suffix operator this rules span names out both as matchers and as grouping keys." + }, + { + "id": "llm_model_attributes_disagree", + "text": "On the same ChatCompletion span, llm.invocation_parameters reports \"model\": \"gpt-4o-mini\" (CrewAI strips a recognised openai/ prefix before calling the SDK) while llm.model_name, read from the response, says openai/gpt-4o-mini. The secondary model keeps anthropic/claude-haiku-4.5 in both. Neither is reliably the id the customer passed. Additionally llm.system is the literal \"openai\" on all 21 LLM spans including the 3 anthropic/claude-haiku-4.5 calls, because it names the CLIENT SDK, not the provider — provider attribution keyed on llm.system is silently wrong. llm.provider is not emitted at all." + }, + { + "id": "no_native_hitl_telemetry", + "text": "CrewAI's HITL (Task(human_input=True) -> HumanInputProvider) emits NO span, NO span event and NO attribute — and crewai_user proves it: the two HITL turns really did gate a destructive tool (turn 5 denied, turn 6 approved) yet their subtrees are indistinguishable in shape from an ordinary tool turn (AGENT + 4 ChatCompletion + 2 delete_file.run). Natively the pause is visible only indirectly: a TOOL span whose output.value is APPROVAL_REQUIRED, then the user's reply reappearing inside a later ChatCompletion span's llm.input_messages.N.message.content as \"User feedback: …\". Nothing in this seed's algebra can detect a HITL gate. CrewAI's own product telemetry does have a `Human Feedback` span (telemetry.py:1134) — reachable only via the leak path above. Earlier revisions of this fixture papered over the gap with a harness `human_approval_gate` span; it was removed on 2026-08-11 because the invisibility is the fixture." + }, + { + "id": "memory_telemetry_is_events_not_spans", + "text": "The instrumentor's four memory wrappers do NOT create spans: _log_span_event (_wrappers.py:340-353) adds a span EVENT named long_term_memory.save / long_term_memory.search / short_term_memory.save / short_term_memory.search to whatever span is current AND flattens the payload onto that span as . attributes (including `value`, `results`, `query` — raw memory content). Not present here twice over: crewai 1.15.12 removed crewai.memory.long_term and .short_term in favour of unified memory, so the wrappers hit the ModuleNotFoundError branch and are never installed. On crewai 1.10.x-era installs these events and attributes appear on AGENT/TOOL spans and would be attributed to whatever span happened to be active." + }, + { + "id": "graph_node_parent_id_is_not_the_topology", + "text": "graph.node.parent_id (4/17 in crewai_agents, 0/25 in crewai_user) is derived from the agent's INDEX IN crew.agents (_find_parent_agent, _wrappers.py:331-338 — 'the previous agent in the list is the parent'), not from execution. The three fanned-out workers are siblings, yet budget_worker reports graph.node.parent_id=weather_worker. It is also absent whenever the agent is first in the list, which is why single-agent crewai_user has none. The real topology is the span parent/child links; graph.node.* must not be used to reconstruct it." + }, + { + "id": "no_automatic_flush_and_no_export_by_default", + "text": "CrewAI registers nothing for customer traces: without the customer's own TracerProvider + exporter, and without an explicit force_flush()/shutdown() before exit, a short-lived CrewAI process exports NOTHING or drops its tail batch. Bounds how much crewai data maple should expect from scripts and CLI usage." + }, + { + "id": "no_harness_spans", + "text": "[H] Both captures are 100% native as of the 2026-08-11 regeneration: 25/25 and 17/17 spans come from openinference.instrumentation.{crewai,openai}, and scenario_a.py / scenario_b.py / common.py contain no tracer, no span, no set_attribute, no set_status and no record_exception (grep-verified). The only harness code left is telemetry.py's TracerProvider + BatchSpanProcessor + OTLPSpanExporter(endpoint) wiring, the CREWAI_DISABLE_TELEMETRY / CREWAI_TRACING_ENABLED env defaults, and force_flush()/shutdown() at exit — all permitted. The prior revision emitted 4 human_approval_gate spans in a `trace-capture.crewai` scope and stamped framework=crewai / capture.id on the resource; both were removed, so the resource is now service.name + telemetry.sdk.* only and the harness bucket is empty in both goldens." + }, + { + "id": "exception_event_is_native", + "text": "The single `exception` span event and the single ERROR status in crewai_agents (fetch_transport_data.run) are NATIVE: _BaseToolRunWrapper calls set_status(ERROR) + record_exception(exception) at _wrappers.py:1080-1084 after starting the span with record_exception=False/set_status_on_exception=False. No scenario code touches it. Unlike google_adk, exception events ARE a usable native CrewAI tool-failure signal — but note CrewAI then swallows the error and feeds 'Error executing tool: …' back to the agent, so the enclosing AGENT and CHAIN spans both end OK." + }, + { + "id": "batching_is_coarse_and_root_arrives_last", + "text": "The BatchSpanProcessor emits one OTLP record per ~2 s flush, each a single resourceSpans block whose scopeSpans array interleaves both instrumentation scopes. Both traces are multi-record and in both the root arrives in the LAST record (root_last: 2) — the CrewAI root ends only when the whole conversation ends, 14.3 s after it started in crewai_user. A root-anchored ingest path would have to buffer the entire conversation." + } + ], + "variants": [ + { + "name": "oi-event-listener", + "trigger": "CrewAIInstrumentor().instrument(use_event_listener=True) — opt-in kwarg, default False (crewai/__init__.py:105). Builds spans from CrewAI's event bus instead of monkey-patching Crew/Task/Tool.", + "effects": "Same scope name and version, DIFFERENT span names and populations: `..execute` replaces `.._execute_core` (_event_listener.py:347-351) and, with create_llm_spans=True (the default in that mode, _event_listener.py:581), the crewai instrumentor emits its own `.llm_call` LLM spans (_event_listener.py:523) — the LLM spans move INTO openinference.instrumentation.crewai and the openai instrumentor becomes optional. Both false_negatives entries about the ChatCompletion population and every span-name statement in this file are variant-specific.", + "exercised_in_captures": false + }, + { + "name": "oi-genai-semconv", + "trigger": "OPENINFERENCE_ENABLE_GENAI_SEMCONV=true (config.py:109, default False at config.py:118); also settable via TraceConfig(enable_genai_semconv=True).", + "effects": "OpenInferenceSpan.end() (_spans.py:73-79) copies derived gen_ai.* aliases (gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.*, gen_ai.provider.name, …) onto every span the OITracer created — i.e. all openinference.instrumentation.crewai spans — at span END. fallback_fingerprints gains the gen_ai.* tier and a generic OTel-GenAI matcher starts firing on CrewAI spans, so priority ordering matters.", + "exercised_in_captures": false + }, + { + "name": "crewai-product-telemetry-adopts-app-provider", + "trigger": "CREWAI_DISABLE_TELEMETRY / CREWAI_DISABLE_TRACKING / OTEL_SDK_DISABLED all unset — THE DEFAULT — combined with the app having installed a real (non-Proxy) TracerProvider before the first Crew/Flow is built. Telemetry.set_tracer() (telemetry/telemetry.py:254-268) then does NOT install its own provider; CrewAI's product-analytics spans are created by the APP's tracer and exported to the app's collector instead of telemetry.crewai.com:4319.", + "effects": "Adds a third scope, `crewai.telemetry` (telemetry/telemetry.py:377 etc.), carrying non-AI product-analytics spans (`Crew Created`, `Crew Execution`, `Task Created`, `Task Execution`, `Tool Usage`, `Tool Usage Error`, `Tool Repeated Usage`, `Human Feedback`, `Flow Creation`, `Flow Execution`, `Environment Context`, `Feature Usage`, deployment spans). It also attaches CommonAttributesSpanProcessor to the APP's provider (telemetry.py:206-236), which stamps `coding_agent` on EVERY span that provider emits — CrewAI spans, HTTP spans, DB spans alike. The fixture disables this (telemetry.py sets CREWAI_DISABLE_TELEMETRY=true before importing crewai), so neither the scope nor the attribute appears in these captures.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "dspy", + "seed": "frameworks/dspy/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.dspy" + }, + "owned_by": "library", + "source": "wire", + "justification": "Derived from the instrumentor package's own module path: DSPyInstrumentor._instrument does trace_api.get_tracer(__name__, __version__, tracer_provider) at openinference/instrumentation/dspy/__init__.py:86-87, where __name__ is literally 'openinference.instrumentation.dspy'. Library-owned, namespaced, and unchangeable by the app (the app can only swap the TracerProvider). Catches 87/87 + 73/73 spans — every span in both captures, which after the 2026-08-11 de-harnessing are 100% native — including the zero-payload adapter and module spans that no attribute rule can reach. This is the ONLY DSPy discriminator that exists — see attr_matchers.why and algebra_violations.", + "priority": 39993 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.dspy" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "C" + }, + { + "key": "user.id", + "authority_predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.dspy" + }, + "validation": [ + "non_empty" + ], + "granularity": "user", + "verdict": "C" + } + ], + "decoy_keys": [ + { + "key": "service.instance.id", + "source": "wire", + "why": "Random uuid4 auto-detected by the OTel Python SDK once per process (sdk/resources/__init__.py:507-528) and stamped on the RESOURCE of every span — 411674cf-... for dspy_user, 8f8942de-... for dspy_agents. Perfectly session-shaped and perfectly stable across a whole capture, which is exactly why it is dangerous: in a script it looks like a session id, in a long-lived server it silently merges every session in the process. Never join on it." + }, + { + "key": "llm.invocation_parameters", + "source": "wire", + "why": "Byte-identical JSON blob ('{\"temperature\": 0.0, \"max_tokens\": 600}') on every LM.__call__ span in a run — a constant that groups like a key but identifies only the sampling config, in the same family as langsmith.trace.session_name." + }, + { + "key": "metadata", + "source": "source_code", + "why": "The third CONTEXT_ATTRIBUTES entry (using_metadata / using_attributes, context_attributes.py:18-26). Written as a JSON blob under the bare key 'metadata'; customers commonly bury a session or thread id inside it. Not a join key: the algebra cannot reach into a JSON value, and there is no agreed inner field name." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "no_token_counts_no_cost_anywhere", + "text": "DSPy spans carry ZERO usage and ZERO cost data. Not one llm.token_count.*, gen_ai.usage.*, cost, or price key exists on any span in dspy_user or dspy_agents (grep-verified across both records.jsonl). This is a source-level property, not a run artifact: openinference-instrumentation-dspy's attribute surface is exactly {span kind, input/output value+mime, llm.model_name, llm.provider, llm.invocation_parameters, llm.input_messages, llm.output_messages, retrieval.documents} (__init__.py:1157-1171) — no usage extractor exists, and the _LMCallWrapper never touches response.usage. Re-verified against the regenerated 2026-08-11 captures. The dialect DOES define the keys and openinference-instrumentation-openai emits them (crewai_user, same corpus), so a DSPy customer who wants token/cost data must install a SECOND instrumentor beneath DSPy (litellm or openai). Maple must expect DSPy traces with complete prompt/completion content and no measurable spend. NOTES.md previously claimed these keys were present; corrected in the same pass." + }, + { + "id": "session_id_is_pure_opt_in", + "text": "Verdict C is unusually strong here. Nothing in DSPy or the instrumentor mints, defaults, or infers a session id — no uuid4 fallback (contrast google_adk), no config-time id (contrast spring_ai's ChatMemory), no per-run id at all. Absent using_session(), a DSPy trace has NO identifier of any granularity: not a session id, not a run id, not an invocation id. dspy.History, DSPy's own conversation abstraction, is a signature INPUT FIELD whose contents are serialised into input.value — carrying the whole conversation and no key to it. The upside is that when the customer does opt in, coverage is total (every span) and root-start-visible." + }, + { + "id": "span_amplification", + "text": "One logical LLM step is 5 spans: .forward -> Predict.forward -> Predict().forward -> .__call__ -> LM.__call__, with only the last carrying model/messages. ReAct multiplies it by iteration count and adds a synthetic finish.__call__ TOOL span per successful run (6 in dspy_user, 3 in dspy_agents) that is control flow, not a tool call. Observed ratio: 87 spans for a 6-turn chat, 73 spans for one research briefing; 15 and 13 LM.__call__ spans respectively, i.e. ~5 spans of overhead per real model call. Any per-span cost model or 'LLM calls' rollup must count openinference.span.kind=LLM only." + }, + { + "id": "signature_names_are_erased", + "text": "Predict span names read Predict(StringSignature).forward, not the customer's signature class name, whenever the signature was built dynamically (dspy.ReAct and dspy.ChainOfThought both do this) — 15/15 and 13/13 occurrences here. Combined with module/tool span names being raw customer class names, span.name in DSPy is neither stable nor framework-identifying in either direction." + }, + { + "id": "adapter_fallback_duplicates_subtrees", + "text": "When ChatAdapter fails to parse a completion, DSPy retries the same logical call through JSONAdapter, so one Predict(...).forward span holds TWO adapter children, each with its own LM.__call__ — a genuine double-charge of the same logical step. Observed during wiring validation (an earlier probe capture held 1 JSONAdapter.__call__ against 6 ChatAdapter.__call__); it did not recur in the 2026-08-11 captures, which are 28/28 ChatAdapter.__call__ and 0 JSONAdapter.__call__. Source-level, not run-specific: dspy/adapters/chat_adapter.py:54-87 constructs a JSONAdapter and retries through it whenever ChatAdapter parsing fails, gated by use_json_adapter_fallback (default True). Dedupe by parent, not by name." + }, + { + "id": "react_swallows_tool_errors", + "text": "The corpus' only DSPy span event is the single `exception` on fetch_transport_data.__call__ (dspy_agents), and it is NATIVE, not [H]: the instrumentor's _ToolCallWrapper opens the span with OTel's default record_exception=True / set_status_on_exception=True (__init__.py:830) and never catches, so the SDK records exception.type/message/stacktrace/escaped and ends the span ERROR (code 2, 'TransportDataError: transport data service unavailable (503)'). It does NOT propagate: dspy.ReAct catches every tool exception and folds 'Execution error in : ...' into its trajectory text, so the parent ReAct.forward and every ancestor end OK. In dspy_agents exactly 1 of 73 spans is ERROR while the run is a business-level failure (a degraded briefing). Worse after de-harnessing: that ERROR span sits in the ORPHAN TransportWorker.forward trace, so the orchestrator trace — the one an operator would open — contains no error at all. Trace-level error rollups keyed on the root will report 100% success for DSPy agents that are failing." + }, + { + "id": "optimizer_traffic_is_indistinguishable", + "text": "dspy.Evaluate and every teleprompter (BootstrapFewShot, MIPROv2, ...) drive dspy.Module.__call__ and dspy.Predict.forward, so an offline compile or eval sweep emits spans identical in scope, name grammar and attribute set to production serving traffic, at orders-of-magnitude higher volume and typically with heavy internal threading (hence the orphan-trace behaviour, at scale). Nothing in the algebra — and nothing on the wire — separates a DSPy optimization run from a DSPy production run. This is the single biggest volume risk for the vendor and it is not addressable by sampling rules maple can write." + }, + { + "id": "thread_fanout_orphans_are_native", + "text": "DSPy fan-out shreds the trace. `ThreadPoolExecutor.submit` does not copy contextvars, and dspy.Parallel/ParallelExecutor copies only its own thread_local_overrides ContextVar (dspy/utils/parallelizer.py) — never the full contextvars.Context — so the OTel current span never reaches the worker thread and each worker's outermost .forward starts a BRAND-NEW parentless trace. The 2026-08-11 dspy_agents capture is the direct evidence: 4 traces for one briefing, 3 of them orphans (61/73 spans), the three worker roots starting within 0.6 ms of each other (real parallelism) and one of them holding the run's only ERROR span. An earlier version of this fixture hid the effect with an explicit otel_context.attach() inside each worker; that was removed per REVIEW_IMPLEMENTATION §2 — the broken shape IS the fixture. Consequences for maple: a DSPy fan-out has no trace-level parent, no session key, and no ordering signal beyond wall-clock; trace-per-request assumptions fail, and the same mechanism applies at scale to dspy.Evaluate and every teleprompter (which thread internally by default)." + }, + { + "id": "scope_version_is_the_instrumentor_not_dspy", + "text": "scope.version is 0.1.38 — openinference-instrumentation-dspy's version, passed as __version__ to get_tracer (__init__.py:86-87). It tracks the instrumentor's release line, not dspy 3.3.0. Nothing on the wire reveals which DSPy version produced the trace, so version-gated ingest behaviour cannot key on scope.version." + }, + { + "id": "all_spans_are_internal_and_ok", + "text": "Every span in both captures is SpanKind INTERNAL (kind=1) — span kind carries no information for DSPy, matching 15/22 corpus frameworks. Statuses are equally uninformative: every wrapper explicitly sets OK (code 1), so 87/87 + 72/73 spans are OK and the lone ERROR is the tool span above. There is no UNSET span anywhere, which is itself a de-harnessing tell — the pre-2026-08-11 captures had 15 UNSET spans and all 15 were fixture-made." + }, + { + "id": "no_container_span_means_no_turn_boundary", + "text": "The instrumentor's outermost span is whatever .forward the app calls first, so there is no session, request, turn or invocation container. dspy_user is 6 independent traces rooted at ChatAssistant.forward, one per turn, and nothing on the wire says they belong together or in what order (start timestamps only). A customer who wants turn boundaries must create them: either a surrounding framework span (HTTP server, task queue) or using_session(). Maple cannot infer a DSPy conversation from the trace graph — there is no edge to infer it from." + } + ], + "variants": [ + { + "name": "openinference-genai-semconv", + "trigger": "OPENINFERENCE_ENABLE_GENAI_SEMCONV=true, or TraceConfig(enable_genai_semconv=True) passed to DSPyInstrumentor().instrument(config=...). Default is FALSE (config.py:109,118 DEFAULT_ENABLE_GENAI_SEMCONV). Not an SDK-version gate — a pure opt-in that any OpenInference-instrumented app can flip process-wide.", + "effects": "OpenInferenceSpan.end() (_spans.py:75-79) dual-writes OTel GenAI attributes derived from the OpenInference ones, only where the key is not already set: gen_ai.operation.name (from openinference.span.kind), gen_ai.provider.name (from llm.provider), gen_ai.conversation.id (COPIED FROM session.id — a second session candidate appears), gen_ai.request.model/max_tokens/temperature (parsed out of llm.invocation_parameters), gen_ai.input.messages / gen_ai.output.messages, gen_ai.tool.* and gen_ai.retrieval.* (_genai_conversion.py:41-57). Adds a gen_ai.operation.name fallback fingerprint that the default variant does not have. Does NOT change scope.name, scope.version, the resource, or classification. All of it is written in end(), so none of it is visible to a head sampler and none of it is on the wire before the span closes.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "effect_ai", + "seed": "frameworks/effect_ai/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "resource", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "telemetry.sdk.name", + "value": "@effect/opentelemetry" + }, + "source": "wire", + "justification": "NATIVE and hardcoded: @effect/opentelemetry Resource.configToAttributes (Resource.js:23-33) always writes telemetry.sdk.name=\"@effect/opentelemetry\" plus telemetry.sdk.language=nodejs|webjs, and notably NO telemetry.sdk.version. Wire-confirmed identical on 100% of resourceSpans blocks in both captures. This is the second known corpus case of a native framework-identifying resource attribute (after mastra's @mastra/otel-exporter). MUST stay sufficient: false: it identifies the process as an Effect app that wired @effect/opentelemetry's NodeSdk/WebSdk — NOT as an AI app and NOT the span as an AI span. An Effect HTTP service with no @effect/ai dependency emits the identical resource, and inside these very captures 10/62 and 8/44 spans under that resource are plain HTTP client spans. It is a useful vendor HINT (it narrows an unknown span to the Effect ecosystem) but never a classifier. Also absent entirely under the otlp-tracer variant.", + "priority": 29983 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "LanguageModel.generateText" + }, + "source": "wire", + "priority": 29982 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "Chat.generateText" + }, + "source": "wire", + "priority": 29981 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "Toolkit.handle" + }, + "source": "wire", + "priority": 29980 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "LanguageModel.streamText" + }, + "source": "source_code", + "priority": 29979 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "LanguageModel.generateObject" + }, + "source": "source_code", + "priority": 29978 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "Chat.streamText" + }, + "source": "source_code", + "priority": 29977 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "Chat.generateObject" + }, + "source": "source_code", + "priority": 29976 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "Chat.export" + }, + "source": "source_code", + "priority": 29975 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "Chat.exportJson" + }, + "source": "source_code", + "priority": 29974 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "EmbeddingModel.embed" + }, + "source": "source_code", + "priority": 29973 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "EmbeddingModel.embedMany" + }, + "source": "source_code", + "priority": 29972 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "PersistedChat.get" + }, + "source": "source_code", + "priority": 29971 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "span.name", + "value": "PersistedChat.getOrCreate" + }, + "source": "source_code", + "priority": 29970 + } + ], + "session_candidates": [], + "decoy_keys": [ + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "Provider generation id (`gen-1785983725-oBoxbOOzp5dMRarKSjdf`), one distinct value per LLM call — 10 distinct in effect_ai_user's single 6-turn session. Session-shaped opaque token; correlates one request." + }, + { + "key": "http.response.header.x-generation-id", + "source": "wire", + "why": "The SAME OpenRouter generation id, duplicated onto the http.client POST span by @effect/platform's blanket response-header capture. Doubly tempting because it appears on a second span type and looks like a cross-span join key; it joins one LLM call to its HTTP span, nothing more." + }, + { + "key": "service.name", + "source": "wire", + "why": "Process identity. Uniquely dangerous in this framework because it is ALSO the instrumentation scope name (internal/tracer.js:273), so it shows up as a per-app constant in two different places and reads like a stable grouping key. It is constant forever for a deployment — joining on it merges every session of every user into one." + }, + { + "key": "effect.fiberId", + "source": "wire", + "why": "SPAN EVENT attribute (never a span attribute) on every Effect.log* event, e.g. `#15`/`#16`/`#17` for the three parallel workers. Identifies a fiber — finer-grained than a run, recycled within a process." + } + ], + "decoy_values": [ + { + "value": "undefined", + "source": "wire", + "why": "The literal 9-character string, not a missing value. Effect's unknownToAttributeValue (internal/utils.js:13-20) falls through to Inspectable.toStringUnknown, so an unset option is serialised as the text `undefined`: 10/10 and 8/8 LanguageModel.generateText spans carry concurrency=\"undefined\" and 10/10 + 6/8 carry toolChoice=\"undefined\". Any future or customer-supplied key routed through Effect's attribute path can be present-but-meaningless in exactly this way, which `present()` and `non_empty` both wave through." + } + ], + "harness_keys_fixture_only": [ + { + "key": "session.id" + }, + { + "key": "capture.id" + }, + { + "key": "framework.name" + }, + { + "key": "conversation.turn" + } + ], + "caveats": [ + { + "id": "single_trace_whole_session", + "text": "effect_ai_user is the ONLY corpus capture in which one trace covers an entire multi-turn session: all 6 turns, 62 spans, one trace id, one root. This is not a framework guarantee — it follows from the scenario running the whole conversation inside one Effect.withSpan in one process. It matters twice: it is why trace id accidentally substitutes for a session key here (see unlinkable_populations), and it is why spans_per_turn for this capture is measured as the user_turn.N SUBTREE size (6/6/6/12/12/19, +1 root = 62) rather than spans-per-trace as in the trace-per-turn frameworks." + }, + { + "id": "failure_repaints_the_whole_ancestry", + "text": "An Effect failure sets ERROR on every enclosing span, not just the one that failed, because each ancestor's Effect also fails. The effect_ai_probe capture (not a golden) shows one provider transport error turning http.client POST, LanguageModel.generateText, Chat.generateText, agent_step.assistant, agent.assistant, user_turn.4 AND the trace root all ERROR from one cause — 7 red spans, one incident. Any error-rate or error-count metric over Effect spans is inflated by trace depth. The inverse also holds: with a tool declared failureMode: \"return\", the failure is caught inside Toolkit.handle, so Toolkit.handle stays OK and only the handler's own span is red — if the customer wrote no handler span, the failure is invisible." + }, + { + "id": "status_ok_is_stamped_explicitly", + "text": "Effect sets StatusCode.OK on EVERY successful span (internal/tracer.js:88-91), so 105/106 spans here have status code 1 rather than the UNSET (0) that most instrumentation leaves. Consumers that treat 'has an explicit status' as a signal, or that count OK-vs-UNSET, will read Effect traces differently from every other framework in the corpus." + }, + { + "id": "attribute_types_are_preserved", + "text": "Unlike spring_ai's Micrometer bridge, Effect preserves OTLP scalar types: intValue/boolValue/doubleValue survive (gen_ai.usage.input_tokens=intValue 138, conversation.approval_gated=boolValue false). But anything non-scalar is stringified by Inspectable.toStringUnknown with 2-space indentation, so arrays arrive PRETTY-PRINTED as JSON text: gen_ai.response.finish_reasons is the string \"[\\n \\\"stop\\\"\\n]\", never an arrayValue. Parsers must handle whitespace inside these values. gen_ai.request.temperature=0 also arrives as intValue, not doubleValue." + }, + { + "id": "span_names_are_module_paths_not_operations", + "text": "GenAI semconv would name the LLM span `chat openai/gpt-4o-mini` with SpanKind.CLIENT; @effect/ai names it `LanguageModel.generateText` with SpanKind.INTERNAL. Every span in both captures is INTERNAL except the 18 http.client POST spans (CLIENT). Span kind is useless for classification here, and any UI that derives an operation label from the span name will show the TypeScript method rather than the model or the tool — the tool name lives in the bare `tool` attribute, not in the `Toolkit.handle` span name." + }, + { + "id": "no_agent_loop_in_the_library", + "text": "LanguageModel.generateText is SINGLE-SHOT: one provider request, tool calls resolved, results NOT fed back. The loop, the agent identity, the worker fan-out and the HITL gate are all caller-written, so two Effect AI apps can produce completely different trace shapes from the same library. This is the root cause of false_negatives entry 1 and of the framework having no session: the library models a call, not a conversation." + }, + { + "id": "hitl_is_simulated_and_the_deny_branch_leaves_no_tool_span", + "text": "[H] @effect/ai has no interrupt / approval / resumable-run mechanism of any kind (no `interrupt`, nothing in Tool/Toolkit). The hitl.* spans are the fixture's. Fixture-shape consequence to keep in mind when reading the capture: in the deny turn the model asked in prose and never called delete_file, so effect_ai_user contains NO failed delete_file span — the denial appears only as hitl.user_decision{approved=false} plus the absence of a tool call, while the approve turn shows the full hitl.request_approval -> tool.delete_file chain." + }, + { + "id": "two_records_per_scenario_by_scope_close", + "text": "Export is BatchSpanProcessor + a scoped flush: NodeSdk.layer's release runs forceFlush() then shutdown() when the program scope closes (NodeSdk.js:21), so there is no shutdown() call to make and no lost-spans-on-exit failure mode. Both captures are exactly 2 records (one 5 s timer tick, one final flush) with the root arriving last in both — the standard root-last pattern, at n=2." + }, + { + "id": "app_spans_are_the_customer_idiom_not_a_fidelity_violation", + "text": "[H] §2 FIDELITY JUDGMENT (2026-08-11, re-review): the 49 harness spans were audited span-population by span-population against 'would an unmodified real customer app plausibly contain this?' and ALL were kept — effect_ai is the corpus case where harness spans and customer app spans are the same artefact. An Effect application IS a tree of Effect.withSpan / Effect.fn spans: the framework creates no root, no agent, no loop, no turn, no session and no HITL, so an app that did NOT write these spans would be a LESS representative fixture, not a cleaner one (strip the roots and Chat.generateText becomes the root — an attribute-less span, see customer_realism_gap). Per population: scenario_*.* roots = the app entrypoint every Effect service has; user_turn.N + session.id = the injection point that IS this framework's only session mechanism (see session.candidates.why); agent.* / agent_step.* = the caller-written loop the library forces (see no_agent_loop_in_the_library); worker.* / orchestrator.fan_out / agent.orchestrator / agent.summary_agent = orchestration the library has no concept of; tool.* = Effect.fn handlers, the default Effect way to write a function, and the reason the failureMode:\"return\" error is visible at all; hitl.* = simulated because the library has no interrupt (see hitl_is_simulated_and_the_deny_branch_leaves_no_tool_span). Verified negative on all four removal triggers: ZERO raw-OTel usage (no @opentelemetry/api import, no getTracer/startSpan/setAttribute/recordException/setStatus/addEvent anywhere under src/), ZERO trace-capture.* scopes, ZERO shape repairs (nothing re-parents, merges or orphan-fixes a native span), and ZERO native-span mutation — wire-confirmed: the complete attribute set of the native spans is LanguageModel.generateText{gen_ai.*, toolChoice, concurrency}, Toolkit.handle{tool, parameters}, Chat.generateText{} with no events on any of them, so all 9 Effect.annotateCurrentSpan calls demonstrably landed on spans the scenario itself opened (harness_mutated_spans: 0)." + }, + { + "id": "harness_writes_gen_ai_request_model_onto_app_spans", + "text": "[H] Fixture artefact kept deliberately, recorded so it is never read as native: 4 harness spans carry the semconv key gen_ai.request.model — scenario_a.user_flow (scenario-a.ts:102) and the three worker.* spans (scenario-b.ts:39) — alongside agent.model. It is plausible customer behaviour (apps do tag their own agent spans with semconv keys) and it is not maple dialect, so §2 does not require its removal, but it has one consequence: this vendor's matchers are span-name-based and correctly ignore all 4 (harness_matched: 0), while an UNKNOWN-TIER rule shaped as key_prefix(\"gen_ai.\") would classify them. The fallback_fingerprints count (gen_ai.operation.name on 10/62 + 8/44, LanguageModel.generateText only) is the native population and is unaffected — operation.name appears on no harness span. Any consumer measuring 'gen_ai spans' by namespace rather than by this key will read 66/62 and 47/44." + }, + { + "id": "harness_spans_are_the_whole_agent_layer", + "text": "[H] 30/62 and 19/44 spans are harness-authored via Effect.withSpan / Effect.fn (agent.*, agent_step.*, worker.*, user_turn.*, tool.*, hitl.*, orchestrator.fan_out, scenario_a.user_flow, scenario_b.orchestration), and both trace roots are among them. Unlike spring_ai's ObservationRegistry spans these are not disguised as framework spans — but they DO land in the framework's scope (there is only one scope), so scope is again no help; provenance is. None of them matches a vendor rule (harness_matched: 0) and none mutates a native span (harness_mutated_spans: 0): every Effect.annotateCurrentSpan call in the scenarios targets a span the scenario itself opened." + } + ], + "variants": [ + { + "name": "otlp-tracer", + "trigger": "App wires @effect/opentelemetry's dependency-free OTLP path (OtlpTracer.layer / Otlp.layer) instead of NodeSdk.layer / WebSdk.layer. No OpenTelemetry SDK is involved; this is the recommended path for non-Node runtimes (workerd, Deno, browser) and for apps that do not want the OTel SDK.", + "effects": "resource_matchers becomes EMPTY: OtlpResource.make (OtlpResource.js:14-33) emits only service.name / service.version / OTEL_RESOURCE_ATTRIBUTES and never sets telemetry.sdk.name, so the one native resource signal disappears. scope.version disappears too — OtlpTracer.js:19-21 builds the scope as `{name: serviceName}` with no version field (the NodeSdk path passes service.version as the scope version). Sampling changes: makeSpan hardcodes `sampled: true` (OtlpTracer.js:58) — there is no Sampler at all, so sampler_surface's OTEL_TRACES_SAMPLER lever does not exist. attr_matchers (span names) are unaffected.", + "exercised_in_captures": false + }, + { + "name": "effect-v4-unstable-ai", + "trigger": "Dependency on effect@4's built-in `effect/unstable/ai` instead of the separate `@effect/ai` package on effect@3 (the line the public Effect docs and context7 already describe).", + "effects": "UNVERIFIED — effect@4 is NOT installed in frameworks/effect_ai (node_modules/effect is 3.22.1 and has no dist/dts/unstable directory), so nothing in this file was checked against it. The AI modules move to new TypeScript module paths, which is exactly what this seed's attr_matchers match on, so every span-name matcher is expected to change. Whether v4 introduces a native conversation/session identifier (which would move session.candidates off empty and the verdict off D) COULD NOT BE DETERMINED from any installed source and must be re-reviewed against an effect@4 capture.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "flue", + "seed": "frameworks/flue/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "@flue/opentelemetry" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded in @flue/opentelemetry 2.0.3 dist/index.mjs:94 (trace.getTracerProvider().getTracer('@flue/opentelemetry')) — package-scoped, library-owned, and the only tracer the adapter ever uses. Catches 25/25 + 21/21 spans, including the source-only span types (flue.coordinator, `flue.operation shell`) that carry no gen_ai.* key. Sufficient because nothing but this adapter emits under an npm-scoped tracer name; the one way to break that is the customer passing their own shared tracer via options.tracer — see variants.custom-tracer-scope and false_positives, in which configuration key_prefix(flue.) is the discriminator.", + "priority": 39992 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "flue." + }, + "source": "wire", + "priority": 29969 + } + ], + "session_candidates": [ + { + "key": "gen_ai.conversation.id", + "authority_predicate": { + "op": "eq", + "key": "flue.operation.kind", + "value": "prompt" + }, + "validation": [ + "non_empty", + "not_in_decoy_values" + ], + "granularity": "session", + "verdict": "A" + }, + { + "key": "flue.instance.id", + "authority_predicate": null, + "validation": [ + "non_empty" + ], + "granularity": "instance", + "verdict": "A" + } + ], + "decoy_keys": [ + { + "key": "flue.session.name", + "source": "wire", + "why": "THE decoy of this framework, and the one that looks most like a session key. In flue_user it is the constant `default` on 25/25 spans — a process-wide literal, not an id. In flue_agents it is `default` on the orchestrator's 4 spans and `task:default:` on the other 17: it names Pi's harness session slot and encodes the delegation tree, changing WITHIN a single trace and turn. Joining on it merges every `default` session of every customer." + }, + { + "key": "flue.parent_session.name", + "source": "wire", + "why": "The other half of the delegation edge; `default` on 17/21 flue_agents spans, absent everywhere in flue_user. Constant-valued, present only on delegated work." + }, + { + "key": "flue.harness.name", + "source": "wire", + "why": "`default` on 46/46 spans — the Pi harness profile name, a build-time constant. Never varies per customer, session or run." + }, + { + "key": "flue.submission.id", + "source": "wire", + "why": "sub_, one per dispatch() delivery — 8 distinct values across the 8 flue_user traces sharing ONE conversation id. Session-shaped and stable within a trace, which makes it the most convincing wrong answer; it is a run id." + }, + { + "key": "flue.operation.id", + "source": "wire", + "why": "op_ per agent operation. Tracks the same granularity as flue.submission.id and is likewise reborn every turn." + }, + { + "key": "flue.task.id", + "source": "wire", + "why": "task_ per subagent delegation, 17/21 flue_agents spans, 4 distinct values in one trace. Correlates a delegate's subtree, not a session." + }, + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "Provider response id on every chat span; unique per model call." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id (call_… / toolu_…). Also stamped on delegated invoke_agent spans as the parent's `task` call id, which makes it look like a parent-child correlation key; it correlates one tool call." + } + ], + "decoy_values": [ + { + "value": "default", + "source": "wire", + "why": "The value of flue.session.name (46/46 spans) and flue.harness.name (46/46). Any candidate resolving to the literal `default` is a harness/session-slot constant, not an identity — listed so not_in_decoy_values rejects it if a future Flue version ever routes it into a candidate key." + } + ], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "session_granularity_rationale", + "text": "Why maple treats gen_ai.conversation.id (not flue.instance.id) as THE session key, per §3's multi-granularity rule. Three granularities are live on every Flue span at once: flue.instance.id (durable agent instance — coarser than a session; a long-lived instance holds many conversations), gen_ai.conversation.id (the conversation — session), and flue.submission.id / flue.operation.id (one delivery — run). The middle one is the session because it is the unit that (a) survives across traces — one id over all 8 flue_user traces — and (b) is what the runtime re-loads history under. flue.instance.id is only session-SHAPED here because each scenario is a single process running a single instance; that coincidence is a fixture property, not a Flue property. The subtlety is that the conversation key is also present at a FINER granularity on the same wire: subagent spans carry their own conv_ ids (5 distinct in the one flue_agents trace), so the key must be read only where flue.operation.kind=prompt. docs/session-identity.md's flue row, which calls flue.instance.id THE session key, is stale and contradicted by this seed." + }, + { + "id": "session_key_arrives_last", + "text": "SILENT-ZERO TRAP. gen_ai.conversation.id is on 100% of spans, but the AUTHORITATIVE population is the root invoke_agent span, which ends last and so exports last: 5/5 multi-batch traces are root-last, and only 4/9 traces (all single-batch) have the session key visible in their first record. An ingest that resolves a trace's session on first sight, and does not revisit, silently un-sessions every trace long enough to span two batches — the more spans a turn has, the likelier it is to be lost. Resolution must be revisitable." + }, + { + "id": "no_gen_ai_system", + "text": "Flue emits NO gen_ai.system. It uses the newer gen_ai.provider.name spelling (=openrouter here) and puts the provider's own model id, slashes intact, in gen_ai.request.model (`anthropic/claude-haiku-4.5`) and the span name. Any cross-framework rule or dashboard keyed on gen_ai.system sees nothing for Flue, and provider attribution must read gen_ai.provider.name." + }, + { + "id": "agent_attribution_is_wrong_on_chat_spans", + "text": "gen_ai.agent.name is the ACTING agent only on invoke_agent spans. On the chat and execute_tool spans nested inside a delegate it reports the registered root agent: all 3 `chat` spans under invoke_agent budget_worker say gen_ai.agent.name=ResearchOrchestrator, and flue.agent.name says the same on 21/21 spans (source: dist/index.mjs:159,194 read event.agentName, which identifiers() also feeds). Per-agent token or latency rollups keyed on gen_ai.agent.name are silently wrong for every delegated model call; use span parentage, flue.task.id or flue.session.name." + }, + { + "id": "content_ships_customer_stack_traces", + "text": "Content-on-by-default is not limited to prompts. The one ERROR span in flue_agents carries an `exception` span event whose exception.stacktrace contains absolute filesystem paths of the running application (/Users/.../frameworks/flue/src/tools.ts:53 and node_modules paths). Errors thrown by customer tool code put their source layout on the wire under the default configuration, gated only by { content: false } — which also removes prompts." + }, + { + "id": "payload_is_mostly_content", + "text": "Content attributes are 84% of all span-attribute bytes in flue_user (76.6 KB of 91.6 KB) and 73% in flue_agents, on conversations of 80-word replies. The single largest attribute observed is a 4,973-byte gen_ai.input.messages; the per-span ceiling is 56 KiB. Volume control for Flue is the content policy, not trace count — and unlike sampling, it is a code change (see content_toggles)." + }, + { + "id": "no_error_status_without_a_thrown_error", + "text": "Status is UNSET (code 0) on 45/46 spans; the single ERROR is the tool that throws. Flue sets error.type + ERROR status + an exception event together in one place (dist/index.mjs:447-462), so present(error.type) and status.code=ERROR are equivalent for Flue — unlike google_adk, where they diverge. The exception event here is NATIVE, not fixture-injected." + }, + { + "id": "harness_resource_attributes", + "text": "[H] The resource attributes framework.name=flue and framework.version=2.0.3 are the FIXTURE's, from src/telemetry.ts:32-33 — not Flue's. Flue contributes no resource attribute at all. Any earlier note treating them as a native resource-level vendor signal (a resource matcher candidate) is wrong; see classification.resource_matchers.why." + }, + { + "id": "dedupe_makes_span_counts_uneven", + "text": "A delegated subagent produces ONE invoke_agent span (from task_start), not two, because dist/index.mjs:116 suppresses the subagent's own operation_start. That is why 4 of the 5 invoke_agent spans in flue_agents lack flue.operation.kind and gen_ai.input/output.messages coverage differs between root and delegate invocations — not a capture gap." + } + ], + "variants": [ + { + "name": "cloudflare-workers-tracing", + "trigger": "Deploying through @flue/vite with the cloudflare target. The generated worker entry calls installDefaultCloudflareTracing() (@flue/vite 2.0.3 dist/index.mjs:266-273) and it is the DEFAULT — includeTracing = tracing !== false (dist/index.mjs:208), i.e. only `tracing: false` in flue.config.ts omits it. This is Flue's own deployed-app path; @flue/opentelemetry is not involved.", + "effects": "Spans are written to the workerd platform tracer (Workers Traces), not to an OTel SDK: there is no instrumentation scope at all, so scope_matchers are dead and attr_matchers are the only classifier. The vocabulary overlaps but is not identical (@flue/runtime dist/tracing-mRWmVbdr.mjs:123-150,326-460): flue.instance.id, flue.submission.id, flue.operation.kind, flue.task.id, flue.tool.origin, flue.turn.purpose, flue.usage.total_tokens and every gen_ai.* key are kept — key_prefix(flue.) still holds — but flue.session.name, flue.parent_session.name, flue.harness.name, flue.agent.name, flue.operation.id, flue.turn.id and flue.event.index DO NOT EXIST, gen_ai.response.finish_reasons is replaced by the singular flue.response.finish_reason, flue.turn.purpose is emitted only when the purpose is not `agent`, and three keys appear that the OTel adapter never emits: flue.span.forced_close, flue.canceled, flue.recovery.operation/.outcome. operation_start returns early for every kind except prompt/skill, so the `flue.operation ` spans the OTel adapter can emit never occur. Session story is unchanged: gen_ai.conversation.id on spans with flue.operation.kind=prompt.", + "exercised_in_captures": false + }, + { + "name": "custom-tracer-scope", + "trigger": "createOpenTelemetryInstrumentation({ tracer }) — a documented option (@flue/opentelemetry dist/index.mjs:94: options.tracer ?? getTracer('@flue/opentelemetry')).", + "effects": "scope.name becomes whatever the app named its tracer, so the scope matcher silently misses every span and classification falls entirely to key_prefix(flue.). Conversely, if the app reuses that tracer for its own spans, the scope stops being single-vendor. No attribute changes.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "google_adk", + "seed": "frameworks/google_adk/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "gcp.vertex.agent" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded in google/adk/telemetry/tracing.py:102 (instrumenting_module_name=\"gcp.vertex.agent\", version=ADK release, schema_url=Schemas.V1_36_0). Library-owned, namespaced, stable across the package. Catches 39/39 + 30/30 spans including the attribute-less `invocation` root, which nothing else can classify. No non-AI span was observed in this scope (see false_positives for the untested co-tenancy caveat).", + "priority": 39991 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "gcp.vertex.agent." + }, + "source": "wire", + "priority": 29968 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.system", + "value": "gcp.vertex.agent" + }, + "source": "wire", + "priority": 29967 + } + ], + "session_candidates": [ + { + "key": "gen_ai.conversation.id", + "authority_predicate": { + "any_of": [ + { + "op": "eq", + "key": "gen_ai.operation.name", + "value": "invoke_agent" + }, + { + "op": "eq", + "key": "gen_ai.operation.name", + "value": "generate_content" + } + ] + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "A" + }, + { + "key": "gcp.vertex.agent.session_id", + "authority_predicate": { + "op": "eq", + "key": "gen_ai.system", + "value": "gcp.vertex.agent" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "A" + }, + { + "key": "gcp.vertex.agent.invocation_id", + "authority_predicate": { + "op": "present", + "key": "gen_ai.request.model" + }, + "validation": [ + "non_empty" + ], + "granularity": "run", + "verdict": "A" + } + ], + "decoy_keys": [ + { + "key": "gcp.vertex.agent.event_id", + "source": "wire", + "why": "Per-ADK-Event UUID, 23/39 and 22/30 spans, 14 distinct values inside the single google_adk_agents trace. Session-shaped (bare uuid4) and adjacent to session_id in the same namespace, but changes several times per turn." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id or ADK-synthesised adk-. Repeats across the HITL ask/deny and ask/approve span pairs, which makes it look like a correlation key; it correlates tool calls, not sessions." + }, + { + "key": "service.name", + "source": "wire", + "why": "Harness-set here [H]; in real deployments it is the app's service name — process identity, never session identity." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "call_llm_generate_content_double_count", + "text": "call_llm and generate_content are near-duplicate nested spans, 1:1 in both captures, with identical durations and overlapping usage attrs. Any per-span LLM-call or token rollup double-counts unless one is dropped — but they are also the seed's two session-candidate populations: dropping call_llm loses the only gcp.vertex.agent.session_id carrier and the content blobs; dropping generate_content loses half the gen_ai.conversation.id carriers. Keep both, dedupe downstream on (gcp.vertex.agent.event_id, parent)." + }, + { + "id": "gen_ai_system_is_not_the_provider", + "text": "gen_ai.system is 'gemini' on generate_content (tracing.py:770, guessed regardless of provider) and the literal 'gcp.vertex.agent' on call_llm. Every request here actually went to openrouter/openai/gpt-4o-mini and openrouter/anthropic/claude-haiku-4.5 via LiteLlm. Provider attribution keyed on gen_ai.system is silently wrong; gen_ai.request.model carries the truth." + }, + { + "id": "execute_tool_placeholder_blobs", + "text": "execute_tool spans hardcode gcp.vertex.agent.llm_request/llm_response to the literal string '{}' (tracing.py:225-226) while call_llm carries the real 1.7-8.6 KB blobs. present() matches both — a value-quality fact, not a classification error (the spans ARE google_adk spans); content-aware consumers must check for the placeholder." + }, + { + "id": "root_anchored_resolution_descends", + "text": "§3's root-most-anchoring must be allowed to descend: the anchor is always a child (invoke_agent), never the trace root — a root-only reader gets zero attributes from 9/9 traces." + }, + { + "id": "harness_injected_exception_event", + "text": "[H] The one exception span event and the one ERROR status (google_adk_agents, execute_tool fetch_transport_data) come from scenario_b.py:82-83 (record_exception/set_status in a BasePlugin on_tool_error_callback). Native ADK sets only error.type=TOOL_ERROR and leaves status UNSET. Do not treat exception events or ERROR status as native ADK signals — use present(error.type). Counted in harness_mutated_spans." + }, + { + "id": "notes_hitl_claim_unverified", + "text": "NOTES.md claims the confirmation handshake is visible inside the llm_request/llm_response blobs; the string adk_request_confirmation occurs nowhere in either capture (nor the pre-prune .bak). The HITL flow IS present but only as 4 execute_tool delete_file spans pairing on gen_ai.tool.call.id — no confirmation span, no confirmation attribute; HITL is invisible to any rule in this seed's algebra." + }, + { + "id": "adk_exports_nothing_by_default", + "text": "Runner-driven ADK honours no OTLP env var — a customer scripting ADK must register their own TracerProvider or nothing is sent. Bounds how much google_adk data maple should expect from non-`adk web` deployments." + } + ], + "variants": [ + { + "name": "adk-schema-v2", + "trigger": "ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN=2; IMPLICIT DEFAULT on Vertex Agent Engine (detected via GOOGLE_CLOUD_AGENT_ENGINE_ID) — telemetry/_schema_version.py:82-90", + "effects": "Replaces the attribute-less `invocation` root with `invoke_workflow {entrypoint}` carrying gen_ai.operation.name=invoke_workflow and gen_ai.conversation.id passed AT SPAN START (node_tracing.py:189-211): the conversation.id candidate gains a carrier population and becomes root-readable, key_at_root_start flips to true, span-name matching on `invocation` breaks. VERIFIED against 2.6.2 source that this is the ONLY behavioural difference: resolve_schema_version() has exactly one call site (_instrumentation.py:70), gating only the root span. call_llm is NOT removed in 2.6.2 (base_llm_flow.py:1388 is unconditional) — its removal is step 2 of an unlanded migration plan; NOTES.md claims it already landed, which is wrong.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "haystack", + "seed": "frameworks/haystack/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "haystack" + }, + "owned_by": "library", + "source": "source_code", + "justification": "NOT observed in these captures — recorded from haystack_integrations/components/connectors/opentelemetry/opentelemetry_connector .py:84, which hardcodes opentelemetry.trace.get_tracer(\"haystack\") — the ONLY library-owned scope name Haystack can produce, and the one the current Haystack docs steer customers to (see the opentelemetry-connector variant). Marked sufficient because it is the sole mechanism that can classify the haystack.agent.step.llm spans once content tracing is off (the default), where they carry zero attributes; a bare product-name tracer is a real if remote collision risk, called out under false_positives. On the enable_tracing() path exercised here the scope is whatever Tracer the app constructed — in these captures the harness's trace-capture.haystack [H], which is DELIBERATELY NOT transcribed: it does not exist in customer data, and no allowlist can enumerate app-chosen names. Classification therefore has to come from attr_matchers.", + "priority": 39990 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "haystack." + }, + "source": "wire", + "priority": 29966 + } + ], + "session_candidates": [], + "decoy_keys": [ + { + "key": "haystack.pipeline.metadata", + "source": "wire", + "why": "The only customer-writable attribute in the dialect and therefore the obvious place an integrator would stash an id — but it is an opaque JSON blob ('{}' on 10/10 pipeline spans here), fixed at Pipeline construction, and shared by every session that pipeline instance serves. Never join on it." + }, + { + "key": "haystack.component.name", + "source": "wire", + "why": "The only human-meaningful identity in the dialect ('assistant', 'orchestrator', 'weather_worker'). It is a static pipeline-topology label — one constant value forever, on 6/38 and 9/42 spans — not an instance identifier. Grouping by it yields one bucket per component for the lifetime of the deployment." + }, + { + "key": "haystack.agent.step", + "source": "wire", + "why": "A per-step counter (0, 1, ...) restarting at 0 inside every agent run; 9/38 and 9/42 spans. Session-shaped only in that it is a small stable-looking scalar." + }, + { + "key": "service.instance.id", + "source": "wire", + "why": "The most session-shaped value in these captures — a bare uuid4, exactly one per capture — and it is neither native Haystack nor a session: the OTel Python SDK mints it per PROCESS in Resource.create(). It looks like a session id only because each fixture scenario is one short-lived process; in a long-lived server it is constant across every session that server ever handles." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [ + { + "key": "capture.id" + } + ], + "caveats": [ + { + "id": "harness_named_scope_native_spans", + "text": "[H] The instrumentation scope in both captures is trace-capture.haystack — a harness-chosen string (frameworks/haystack/telemetry.py:48). It is NOT evidence of harness provenance: every one of the 80 spans is emitted by haystack-ai's own tracing calls through opentelemetry-haystack's OpenTelemetryTracer, and the harness contributes only the tracer NAME. Provenance grep over frameworks/haystack/*.py finds zero start_as_current_span, zero set_attribute, zero record_exception, zero set_status, zero add_event outside telemetry.py's provider wiring — hence harness: 0 and harness_mutated_spans: 0 in the goldens. The inverse of the spring_ai case (native-looking scope, harness spans): here the scope name is the only harness artifact and the spans under it are entirely native. It is deliberately absent from scope_matchers." + }, + { + "id": "values_are_not_all_strings", + "text": "NOT all attribute values are strings — haystack/tracing/utils.py coerce_tag_value() passes bool/str/int/float through UNTOUCHED and only JSON-encodes everything else. On the wire that means intValue for haystack.agent.step, haystack.agent.steps_taken, haystack.agent.max_steps, haystack.component.visits and haystack.pipeline.max_runs_per_component, and doubleValue for haystack.agent.step.tool.output whenever a tool returns a number (haystack_agents: calculate -> 750). A consumer that assumes stringValue drops or mistypes them; under the canonical stringification in scripts/verify-seed.ts they compare as '1' / '750'." + }, + { + "id": "pipeline_output_data_is_always_empty", + "text": "haystack.pipeline.output_data is the literal '{}' on 10/10 pipeline spans in both captures, and always will be: pipeline.py:385-393 passes the still-empty pipeline_outputs dict in the tag map, and OpenTelemetryTracer.trace() serializes the tags at span OPEN, before the pipeline has run. Same for haystack.pipeline.metadata='{}' when no metadata is set. A google_adk-style placeholder blob: present() matches, the content is worthless. The real outputs are on the component/agent spans." + }, + { + "id": "root_is_not_findable_by_name", + "text": "Span names are static and non-unique within a trace: the single haystack_agents trace contains FOUR haystack.pipeline.run spans (1 root + 3 worker sub-pipelines invoked as PipelineTools), 9 haystack.component.run, 9 haystack.agent.step and 9 haystack.agent.step.llm. All identity is in attributes. Root detection must be structural." + }, + { + "id": "content_encoding_is_value_dependent", + "text": "Content tags are JSON strings EXCEPT where the payload contains a non-JSON-serialisable object, in which case coerce_tag_value falls back to Python repr(). Measured: 16/18 haystack.agent.step.llm.input values are JSON and 2/18 are repr (the haystack_agents orchestrator, whose tool list holds PipelineTool objects); 4/5 haystack.agent.tools are JSON and 1/5 is repr; llm.output is JSON 18/18. Parsers must tolerate both, per-value — it is not a fixed property of the key." + }, + { + "id": "no_error_signal_of_any_kind", + "text": "hasError is false on 80/80 spans and status is UNSET on 80/80, including the always-failing fetch_transport_data 503 in haystack_agents. Failure surfaces only as the string {\"error\": \"...\"} inside the content-gated haystack.agent.step.tool.output. Combined with the missing status/event surface in the Span ABC, Haystack cannot report an error to a tracing backend at all." + }, + { + "id": "hitl_is_native_but_invisible", + "text": "Scenario A exercises Haystack's first-class ConfirmationHook/before_tool gate (only the stdin UI is substituted). It produces NO telemetry: the approval decision gets no span and no event, and a REJECTED tool call produces no haystack.agent.step.tool span at all, because the hook strips the call before the execution stage. Wire-confirmed: the denied delete_file(/tmp/report-final.txt) turn has 2 agent steps and zero tool spans, while the approved delete_file(/tmp/scratch-notes.txt) turn has 1 tool span. HITL is invisible to every rule in this seed's algebra; the only trace of a denial is a sentence inside a content blob." + }, + { + "id": "harness_class_names_leak_into_native_values", + "text": "[H] haystack.component.fully_qualified_type on the briefing_notes span reads 'scenario_b.BriefingNotes' — a harness-authored component class. The SPAN is native (emitted by PipelineBase._create_component_span), so it is not harness by provenance; only the value names fixture code. Worth flagging because fully_qualified_type is otherwise the best available signal for what a component actually is, and in customer data it will name customer classes just as often." + }, + { + "id": "parallel_fanout_is_real_and_context_propagates", + "text": "Verified from the wire, not prose: the three worker tool spans in haystack_agents start within 0.7 ms of each other and overlap for seconds (2088 / 5020 / 4743 ms durations under one 7592 ms step span). Agent dispatches tool calls through a ThreadPoolExecutor bounded by tool_concurrency_limit (default 4) and propagates context with contextvars.copy_context(), so the child spans nest correctly across the thread boundary — no orphans, no split traces." + }, + { + "id": "execution_mode_records_the_sync_async_path", + "text": "haystack.pipeline.execution_mode is 'sync' on 10/10 pipeline spans here. AsyncPipeline takes a parallel code path (pipeline.py:906, agent.py:1203) with the same span names and the same tag vocabulary, so nothing in this seed changes — but it is the attribute to check when reconciling a customer's async deployment." + } + ], + "variants": [ + { + "name": "content-tracing-disabled", + "trigger": "HAYSTACK_CONTENT_TRACING_ENABLED unset or != 'true' — THIS IS THE DEFAULT. haystack/tracing/tracer.py:117 reads it exactly once, in ProxyTracer.__init__ at `import haystack.tracing`; setting it later is silently ignored (the documented after-import escape hatch is assigning haystack.tracing.tracer .is_content_tracing_enabled = True). The captures deliberately run with it ON.", + "effects": "Drops all 8 set_content_tag keys — haystack.{component,agent}.{input,output}, haystack.agent.step.llm.{input,output}, haystack.agent.step.tool.{input,output} — i.e. 82.4% (haystack_user) / 88.1% (haystack_agents) of all attribute bytes. Two answers in this file change: (1) haystack.agent.step.llm spans (9/38 + 9/42) become ATTRIBUTE-LESS, so attr_matchers no longer classify them at all (agent.py:1162 opens that span with no tags); (2) model id, finish_reason and token usage — which exist ONLY inside the haystack.agent.step.llm.output JSON blob — disappear from the wire entirely, as do tool errors. No other span type loses its classifiability: every other span keeps at least one ungated haystack.* tag.", + "exercised_in_captures": false + }, + { + "name": "opentelemetry-connector", + "trigger": "The app adds haystack_integrations.components.connectors.opentelemetry .OpenTelemetryConnector as a pipeline component instead of calling haystack.tracing.enable_tracing() itself. This is the path docs.haystack.deepset.ai /docs/tracing now points at, because Haystack 2.x's auto-enable-if-opentelemetry-sdk -is-installed behaviour was REMOVED in 3.0.", + "effects": "The only variant that changes the scope answer: opentelemetry_connector.py:84 does get_tracer(\"haystack\"), so scope.name becomes the library-owned literal 'haystack' (still no scope.version, no schemaUrl) instead of an app-chosen string. Adds one extra haystack.component.run span for the connector component itself. Attribute dialect is unchanged.", + "exercised_in_captures": false + }, + { + "name": "openinference-haystack", + "trigger": "The app installs openinference-instrumentation-haystack and calls HaystackInstrumentor().instrument(). This is what BOTH mainstream third-party backend guides prescribe for Haystack — signoz.io/docs/haystack-monitoring and langfuse.com/integrations/frameworks/haystack — neither mentions the native opentelemetry-haystack path at all.", + "effects": "Nothing in this file applies. Scope becomes openinference.instrumentation.haystack, the dialect becomes openinference.span.kind + llm.* + input.value/output.value, and the span names/tree come from wrapping Pipeline.run / Pipeline._run_component / Component.run rather than from Haystack's own tracing calls. It is a different vendor entry, not a variant of these rules — recorded here because a haystack customer arriving at maple is arguably MORE likely to be on this path than on the one these captures exercise.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "langchain", + "renamed_from": "langgraph", + "seed": "frameworks/langgraph/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "resource", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "langsmith.internal_provider", + "value": "true" + }, + "source": "source_code", + "justification": "Set ONLY by langsmith's own fallback TracerProvider (_otel_client.py:105-110, literal comment 'Marker to identify LangSmith's internal provider'). Genuinely library-owned and framework-identifying, and it is a whole PROCESS-level marker — but sufficient stays false because that provider is a LangSmith-generic provider: a plain LangChain or bare @traceable app produces the identical resource, so it identifies the LangSmith dialect, not LangGraph. source: source_code — NOT observable in these captures (the scenarios install their own provider first, so the resource here is stock OTel-Python telemetry.sdk.* plus app-chosen service.name/version/deployment.environment [H]).", + "priority": 29965 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "langsmith" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded: trace.get_tracer(\"langsmith\", tracer_provider=...) at _otel_exporter.py:243 — one call site, one tracer for the whole package, no version and no schema_url argument (wire confirms scope.version = \"\" and no schemaUrl on 59/59 spans). It is a bare, unnamespaced word, but it is a product name rather than a generic term like 'ai'/'gen_ai', and nothing else in the OTel ecosystem claims it, so the collision risk is judged low enough for sufficiency. Two things temper that: (a) with no version and no schema_url there is NO secondary confirmation available for a tiebreak, unlike google_adk; (b) the scope is the LangSmith dialect's, not LangGraph's — plain LangChain and bare @traceable spans land in it too (see false_positives). Coverage-wise sufficiency is free: attr_matchers already catch 59/59 spans independently, so downgrading to sufficient:false would not lose a single span.", + "priority": 39989 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "langsmith." + }, + "source": "wire", + "priority": 29964 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.system", + "value": "langchain" + }, + "source": "wire", + "priority": 29963 + } + ], + "session_candidates": [ + { + "key": "langsmith.metadata.thread_id", + "authority_predicate": null, + "validation": [ + "non_empty", + "not_in_decoy_values" + ], + "granularity": "session", + "verdict": "B" + } + ], + "decoy_keys": [ + { + "key": "langsmith.trace.session_name", + "source": "wire", + "why": "THE prime decoy of this framework: 'session' here is LangSmith's legacy word for a tracing PROJECT. The value is $LANGSMITH_PROJECT (default 'default'), a deployment-wide constant that never changes for the life of the process — 42/42 spans hold langgraph_user and 17/17 hold langgraph_agents. Joining on it merges every trace the deployment ever produced into one 'session'." + }, + { + "key": "langsmith.trace.session_id", + "source": "source_code", + "why": "Same thing as session_name, as a UUID: the LangSmith project id (_otel_exporter.py:97,552, set from run_info.session_id). Not observed here because OTEL-only mode never talks to the LangSmith API and so never learns a project id — but a hosted/hybrid customer emits it, and it looks exactly like a session uuid." + }, + { + "key": "langsmith.metadata.LANGSMITH_PROJECT", + "source": "wire", + "why": "The same project name again, arriving by a different route: every LANGCHAIN_* / LANGSMITH_* environment variable is mirrored into metadata (langsmith/env/_runtime_env.py:171-199, minus an exclusion list and any name containing key/secret/token). Constant per deployment. 42/42 + 17/17." + }, + { + "key": "langsmith.metadata.checkpoint_ns", + "source": "wire", + "why": "Looks session-shaped (:) and sits in the same namespace as thread_id, but it is the per-node-task checkpoint namespace: 11/42 + 11/11 spans, 11 distinct values inside langgraph_user and 5 inside the SINGLE langgraph_agents trace. Promoted by an ensure_config allowlist (runnables/config.py:297) that also covers 'model'. Present even when the customer passes no configurable at all." + }, + { + "key": "langsmith.metadata.langgraph_checkpoint_ns", + "source": "wire", + "why": "Same value as checkpoint_ns on the node-span population (34/42 + 16/17, 12 and 5 distinct values). Changes several times per turn; correlates a Pregel task, never a conversation." + }, + { + "key": "langsmith.metadata.revision_id", + "source": "wire", + "why": "Deployment identity, not session identity: $LANGCHAIN_REVISION_ID or, by default, `git describe --tags --always --dirty` of the process CWD (_runtime_env.py:193-201). Constant for the life of a deploy; 04d3fbf-dirty on 59/59 spans here. Also a leak: it publishes the host repo's git state." + }, + { + "key": "service.instance.id", + "source": "wire", + "why": "OTel-SDK-minted uuid4, one per PROCESS (resource attribute). In a one-scenario-one-process fixture it aliases the session perfectly and in a long-lived server it aliases nothing. Never join on it." + } + ], + "decoy_values": [ + { + "value": "default", + "source": "source_code", + "why": "LangSmith's default project name when neither LANGSMITH_PROJECT nor LANGCHAIN_PROJECT nor LANGCHAIN_SESSION is set (utils.py:446-462). It is therefore the most common value of langsmith.trace.session_name and langsmith.metadata.LANGSMITH_PROJECT in the wild. Not observed here (the harness sets the project to the capture id [H]), and listed as a value so that any future candidate carrying it is forced to state 4 rather than 6." + } + ], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "two_stage_flush_or_nothing", + "text": "A plain provider.force_flush() exports NOTHING. langsmith converts runs into spans on its own background worker, so at process exit the TracerProvider's queue is still empty. The capture only works because common.py:flush_tracing() first calls langchain_core.tracers.langchain.wait_for_all_tracers() and get_client().flush(), and only then force_flush()+shutdown(). Any customer (or maple onboarding doc) that follows the standard OTel shutdown recipe gets an empty pipeline with no error — a silent-zero trap that applies to every LangChain-based integration, not just LangGraph." + }, + { + "id": "metadata_namespace_is_an_open_door", + "text": "langsmith.metadata.* is NOT a fixed key set. Two blanket loops feed it: (1) langchain-core copies every scalar entry of the `configurable` dict into inheritable LangSmith metadata — the only exclusions are keys starting with '__' and the single literal 'api_key' (runnables/config.py:151-167), so ARBITRARY customer-chosen keys (user ids, tenant ids, emails, anything) arrive as span attributes on every span of the tree; (2) langsmith mirrors every LANGCHAIN_*/ LANGSMITH_* environment variable into metadata, minus a small exclusion list and any name containing key/secret/token (_runtime_env.py:171-199). Consequences: maple must treat langsmith.metadata.* as unbounded-cardinality customer data (PII risk, attribute-count blowup), and the seed's session key is a customer-namespace key that merely happens to be conventional — a customer passing configurable.thread_id with non-session semantics would poison it. The upside is that the same mechanism is exactly why thread_id reaches 42/42 spans including the root." + }, + { + "id": "gen_ai_system_is_guessed_from_the_model_string", + "text": "_set_gen_ai_system() substring-matches the model name against a hardcoded table (anthropic/claude, gpt/openai, gemini, mistral, groq, ...), defaulting to 'langchain'. Every request in these captures went to OpenRouter, yet llm spans report gen_ai.system=openai (5) and anthropic (3). Provider attribution keyed on gen_ai.system is silently wrong; gen_ai.request.model carries the truth (openai/gpt-4o-mini, anthropic/claude-haiku-4.5)." + }, + { + "id": "interrupts_are_error_spans", + "text": "LangGraph's native HITL gate (interrupt() + Command(resume=...)) produces no dedicated span, attribute or event. The interrupted tool_executor span simply carries status.code=2 and an `exception` event with exception.type='Exception' and the GraphInterrupt repr in exception.message. 2 of the 3 ERROR spans in the corpus are successful approval pauses. Also: a DENIED tool never produces a tool span at all — the denial exists only inside the parent node's gen_ai.completion blob." + }, + { + "id": "one_invoke_one_trace_multiplies_traces_per_session", + "text": "Each graph.ainvoke() is its own trace, and an interrupt/resume pair splits one logical turn into two traces: 6 user turns -> 8 traces in langgraph_user, all joined only by langsmith.metadata.thread_id. Cost/quota reasoning must not equate traces with turns, and any per-trace head sampler shreds a session geometrically (a 6-turn session survives intact with probability p^8)." + }, + { + "id": "no_native_span_or_trace_identity_attributes", + "text": "langsmith.trace.id, langsmith.span.id and langsmith.span.dotted_order are absent from the wire — identity is carried purely by native OTel traceId/spanId/parentSpanId. There is consequently NO run-granularity join key of any kind in this dialect (no invocation id, no run id), which is why this seed has exactly one session candidate and no run-granular one." + }, + { + "id": "span_kind_and_names_are_useless_for_classification", + "text": "59/59 spans are INTERNAL (kind=1). Span NAMES are pure customer input — they are the graph node names (assistant, tool_executor, route, orchestrator, budget_worker) and the compiled graph name; without builder.compile(name=...) the root is the literal string 'LangGraph'. No span-name predicate may be written for this framework. The type discriminator is the attribute langsmith.span.kind (chain|llm|tool observed; retriever|prompt|parser|embedding also exist in source)." + }, + { + "id": "attribute_values_are_properly_typed", + "text": "Unlike spring_ai, this exporter preserves OTLP types: gen_ai.usage.* arrive as intValue, langsmith.metadata.ls_temperature as doubleValue, langsmith.metadata.stream as boolValue. Canonical stringification therefore matters for eq() predicates over these keys (bool -> \"true\"/\"false\")." + }, + { + "id": "stale_probe_capture_present", + "text": "captures/langgraph_smoke (1 record, 3 spans, 1 trace) is a throwaway OTel-wiring probe from the same review. It is deliberately NOT listed in captures: above and must never be counted as fixture data; §7 regeneration should delete it." + } + ], + "variants": [ + { + "name": "langsmith-internal-provider", + "trigger": "No non-proxy global TracerProvider installed before langsmith.Client() is constructed (client.py:1377-1394 reuses an existing one). This is the DEFAULT for any app that just sets LANGSMITH_TRACING/LANGSMITH_OTEL_ENABLED without building an OTel SDK provider of its own — langsmith then builds one itself (_internal/otel/_otel_client.py:105-114).", + "effects": "Resource changes, nothing else: it gains the boolean langsmith.internal_provider=true and service.name defaults to the literal 'langsmith' (OTEL_SERVICE_NAME override). resource_matchers gains a genuinely framework-identifying, near-sufficient predicate that these captures cannot show. Export also defaults to $LANGSMITH_ENDPOINT/otel (LangSmith cloud) rather than the customer's collector, so this variant is common in the wild but rarely reaches a third-party backend. Span attributes, scope and session behaviour are identical.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "litellm", + "seed": "frameworks/litellm/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "resource", + "sufficient": false, + "predicate": { + "op": "present", + "key": "model_id" + }, + "source": "wire", + "justification": "`model_id` is a NON-STANDARD resource key written only by LiteLLM's own resource builder (_get_litellm_resource, opentelemetry.py:364-374, defaulting to OTEL_MODEL_ID or, failing that, service.name). It is the only native framework-identifying resource signal in the captures — present on 6/6 resourceSpans across both captures alongside deployment.environment (LiteLLM's OTEL_ENVIRONMENT_NAME, default 'production'). It CANNOT be sufficient for two reasons. (1) It is a process-wide resource: LiteLLM builds and installs the GLOBAL provider, so every span any other library in the same process emits inherits model_id too — a sufficient rule would claim the whole process. (2) It is environment-dependent: _init_tracing REUSES an already-installed global SDK TracerProvider (opentelemetry.py:549-557), so whenever another framework wires OTel first — the normal passthrough case — LiteLLM's spans carry that app's resource and no model_id at all.", + "priority": 29962 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "litellm" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded in litellm/integrations/opentelemetry.py:70 (LITELLM_TRACER_NAME) and again in the v2 rewrite at otel/logger.py:69. It is the ONLY thing that classifies `raw_gen_ai_request` (17 spans across the two captures), which carries no gen_ai.*, no litellm.* and no stable attribute namespace at all — see algebra_violations[0]. Catches 18/18 + 16/16 spans: after the 2026-08-11 fidelity pass the fixture emits nothing of its own, so every span in both golden captures is a native LiteLLM span in this one scope. Two caveats that argue for a low priority rather than for sufficient: false. (a) The name is a bare word and is overridable by the app via OTEL_TRACER_NAME (opentelemetry.py:70), so a lookup can both collide and be evaded. (b) In proxy deployments and under otel-v2 this scope also carries non-AI spans (see false_positives) — they are still LiteLLM's spans, so vendor attribution stays correct even when 'is this an AI span' does not.", + "priority": 39988 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "litellm." + }, + "source": "wire", + "priority": 29961 + } + ], + "session_candidates": [ + { + "key": "metadata.user_api_key_end_user_id", + "authority_predicate": { + "op": "present", + "key": "litellm.call_id" + }, + "validation": [ + "non_empty", + "not_in_decoy_values" + ], + "granularity": "user", + "verdict": "A" + }, + { + "key": "metadata.user_api_key_user_id", + "authority_predicate": { + "op": "present", + "key": "litellm.call_id" + }, + "validation": [ + "non_empty", + "not_in_decoy_values" + ], + "granularity": "user", + "verdict": "A" + } + ], + "decoy_keys": [ + { + "key": "service.instance.id", + "source": "wire", + "why": "RESOURCE attribute, a fresh uuid4 per PROCESS generated by the OTel Python SDK (sdk/resources/__init__.py:507-528), not by LiteLLM. Because each scenario is one process it takes exactly one value per capture and looks like a perfect session id; in a long-lived gateway it is one value for millions of sessions. The archetypal fixture-shaped trap: predicates fall through to resource attributes, so a candidate on this key would resolve on EVERY span." + }, + { + "key": "model_id", + "source": "wire", + "why": "RESOURCE attribute; defaults to OTEL_MODEL_ID or, failing that, service.name (opentelemetry.py:254-255, 370) — here the literal 'litellm-user' / 'litellm-agents'. Deployment identity, constant forever." + }, + { + "key": "deployment.environment", + "source": "wire", + "why": "RESOURCE attribute from OTEL_ENVIRONMENT_NAME, default 'production' (opentelemetry.py:253). A process-wide constant." + }, + { + "key": "litellm.call_id", + "source": "wire", + "why": "A fresh uuid4 per completion() call (17 distinct values across 17 spans). Correlates one LLM request, not a conversation — and it is the authority predicate for the candidates above precisely because it marks the request population." + }, + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "Provider response id (gen-1785983497-…), unique per call." + }, + { + "key": "llm.openrouter.id", + "source": "wire", + "why": "The same provider response id repeated on raw_gen_ai_request under the provider-prefixed namespace. Unique per call." + }, + { + "key": "metadata.user_api_key_hash", + "source": "wire", + "why": "Hash of the virtual key. Key identity, not session identity — and empty in SDK mode like the other 29 blank metadata.* keys." + }, + { + "key": "metadata.requester_metadata", + "source": "wire", + "why": "The only channel for application metadata, and it arrives as a stringified PYTHON DICT (single quotes), not JSON. A customer will inevitably hide their own session id in here; it is not parseable by the algebra and must never be treated as a key." + } + ], + "decoy_values": [ + { + "value": "", + "source": "wire", + "why": "30 of the 33 metadata.* keys on EVERY litellm_request span are the empty string in SDK mode (only applied_guardrails, requester_metadata and usage_object hold data). present() matches all of them, so any presence-only session resolution collapses every LiteLLM trace in the fleet onto one \"\" group. This is the single most important validation fact for this vendor." + }, + { + "value": "default_user_id", + "source": "source_code", + "why": "LiteLLM's proxy sentinel for the global admin (proxy/_types.py:3106). Not observed here (SDK mode blanks the key), but a proxy deployment stamps it into metadata.user_api_key_user_id for every admin-key request, producing a deployment-wide constant that looks like a real user id." + } + ], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "litellm_is_usually_a_passthrough", + "text": "The most important fact about this vendor: LiteLLM is a gateway, not an agent framework, so most real LiteLLM telemetry arrives UNDERNEATH another framework's spans rather than on its own. Expect litellm to co-occur with a second AI vendor in the same trace far more often than it appears alone, and expect its spans to be leaves. Vendor must be per-span; a per-trace vendor scalar is wrong for every passthrough deployment." + }, + { + "id": "silent_attribute_loss_landmine", + "text": "SOURCE-VERIFIED, opentelemetry.py:1206 (sync) and 1936 (async): should_create_primary_span = parent_span is None or get_secret_bool('USE_OTEL_LITELLM_REQUEST_SPAN'). With a parent span present and the flag unset — the DEFAULT for any wrapper — LiteLLM creates no span and stamps every gen_ai./llm./litellm./metadata. attribute onto the caller's span (line 1224). If that caller's span has already ended (LiteLLM's success callback runs after a synchronous `with` block exits) the OTel SDK discards each set_attribute with a 'Setting attribute on ended span' log line and nothing else. The first run of this fixture produced 28 spans with ZERO token, model or message data and no error. Any framework that wraps LiteLLM without setting this flag exports LLM spans with no LLM attributes — a silent-zero trap maple should expect to see in the wild." + }, + { + "id": "cost_is_only_inside_a_json_string", + "text": "There is no gen_ai.usage.cost attribute. Per-call cost exists on the trace ONLY as `response_cost` inside the hidden_params JSON string attribute (e.g. {\"model_id\": null, …, \"response_cost\": 0.00011625, …}, ~1.4 KB), and a second time inside metadata.usage_object / llm..usage, both of which are Python repr() with single quotes rather than JSON. First-class cost exists only on the opt-in gen_ai.client.token.cost metric." + }, + { + "id": "thirty_of_thirty_three_metadata_keys_are_blank", + "text": "Every litellm_request span carries exactly 33 metadata.* attributes of which 30 are the empty string in SDK mode (user_api_key_hash, _team_id, _org_id, _user_id, _end_user_id, spend_logs_metadata, team_alias, …). They are pure noise — about half the attribute count of the corpus' widest span — but always present, so present()-based rules and non-null-based dashboards both see them as populated." + }, + { + "id": "six_attribute_families_on_one_span", + "text": "A single litellm_request span carries 50-63 attributes spanning six vocabularies at once: OTel gen_ai semconv, Traceloop/OpenLLMetry llm.*, INDEXED gen_ai.completion..function_call.*, LiteLLM's own litellm.*, 33 metadata.*, and the bare unnamespaced hidden_params. Widest single span in the corpus; any family-based classifier will see several families fire simultaneously." + }, + { + "id": "span_names_are_customer_overridable", + "text": "metadata={'generation_name': …} replaces the span name for BOTH litellm_request (opentelemetry.py:2718-2722) and raw_gen_ai_request (opentelemetry.py:1306). Combined with the gen-ai-latest-experimental rename, no span-name predicate is safe for this vendor — which is why none appears in this seed despite `litellm_request` being an unusually distinctive name." + }, + { + "id": "scope_name_is_env_overridable", + "text": "LITELLM_TRACER_NAME = os.getenv('OTEL_TRACER_NAME', 'litellm') (opentelemetry.py:70) — read at MODULE IMPORT, so the scope name a customer's spans arrive under is app-controllable. The default is a bare, unversioned, schema-less word: scope.version and scope.schemaUrl are both empty on all 34 native spans because get_tracer is called with the name alone (opentelemetry.py:561). otel-v2 passes a version; v1 never does." + }, + { + "id": "gen_ai_system_is_the_provider_not_litellm", + "text": "gen_ai.system is `openrouter` on all 17 litellm_request spans — the routed provider, never `litellm`. gen_ai.request.model is the provider-side name (openai/gpt-4o-mini) while litellm.provider.model is the routed LiteLLM name (openrouter/openai/gpt-4o-mini); the two differ on every span. LiteLLM self-identifies in exactly one place in the whole integration: gen_ai.framework=\"litellm\" on the opt-in METRICS (opentelemetry.py:1471). No trace attribute names the framework." + }, + { + "id": "operation_name_is_not_a_semconv_value", + "text": "gen_ai.operation.name = `acompletion` (the litellm call type: completion / acompletion / …), not `chat`. It only becomes `chat` under the gen-ai-latest-experimental opt-in. NOTES.md previously claimed `chat` for the default dialect; the wire disagrees and the wire wins. Any unknown-tier rule that switches on the VALUE of gen_ai.operation.name will not recognise default-dialect LiteLLM." + }, + { + "id": "values_are_python_repr_not_json", + "text": "gen_ai.input.messages / gen_ai.output.messages / hidden_params are real JSON, but metadata.usage_object, metadata.requester_metadata and every llm..* blob are Python str()/repr() with single quotes and None/True/False literals. A JSON parser fails on them. All attribute values arrive as OTLP stringValue, including counts and booleans (llm.is_streaming=\"False\")." + }, + { + "id": "native_spans_are_always_ok", + "text": "Both native span types set StatusCode.OK explicitly (opentelemetry.py:1282, 1313) rather than leaving UNSET. Wire-confirmed after the fidelity pass: 34/34 spans carry status code 1 and 0 span events — including litellm_agents, whose fetch_transport_data tool raised a 503 and whose model wrote about the failure in prose. There is NO error signal of any kind on a LiteLLM trace when the APPLICATION fails; only a failed LLM call reaches _handle_failure, which no scenario exercised. otel-v2 changes the success status to UNSET." + }, + { + "id": "default_topology_is_one_trace_per_request", + "text": "WIRE-VERIFIED 2026-08-11, the headline fact for maple. A customer calling litellm.acompletion() with no other instrumentation gets ONE INDEPENDENT TRACE PER HTTP REQUEST — 2 spans, root litellm_request, child raw_gen_ai_request, nothing linking it to the previous call. A 6-turn chat is 9 disconnected traces (litellm_user); a 5-agent orchestration with a real parallel fan-out is 8 disconnected traces (litellm_agents). Combined with the total absence of a session key, LiteLLM is the corpus' worst case for conversation reconstruction: neither a trace id nor an attribute joins two calls of the same conversation. Any grouping maple offers here must come from resource identity + time, or from the customer adding their own parent span — which then triggers the no-primary-span landmine." + }, + { + "id": "harness_no_longer_emits", + "text": "PROVENANCE, post-§7: the fixture (frameworks/litellm/**) creates no spans and sets no attributes. telemetry.py wires LiteLLM's own OTel callback and calls force_flush; agent.py/scenario_*.py call acompletion() and nothing else, and no USE_OTEL_LITELLM_REQUEST_SPAN or content-capture override is set. The one fixture-visible value on the wire is metadata.requester_metadata, which is LiteLLM's own application-metadata channel (metadata={'metadata': {...}}), not instrumentation — a real customer's app metadata lands in exactly the same place. captures/litellm_semconv_probe still contains 4 pre-pass harness spans and is excluded from the goldens." + } + ], + "variants": [ + { + "name": "gen-ai-latest-experimental", + "trigger": "OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental (or the programmatic OpenTelemetryConfig.semconv_stability_opt_in). Unset by default; parsed once in OpenTelemetryConfig.__post_init__ (opentelemetry.py:258).", + "effects": "Span name becomes `{operation} {model}` instead of the literal `litellm_request` (opentelemetry.py:2718-2731) — so eq(span.name, litellm_request) breaks; SpanKind becomes CLIENT instead of INTERNAL (opentelemetry.py:1279-1280); the `raw_gen_ai_request` child is SUPPRESSED entirely (opentelemetry.py:1298-1300), removing half the span population and every llm..* key; gen_ai.system is replaced by gen_ai.provider.name, so any eq(gen_ai.system, …) rule stops firing; gen_ai.operation.name changes VALUE from `acompletion` to the semconv `chat`; llm.is_streaming disappears. UNCHANGED and therefore load-bearing for classification: scope.name=litellm, litellm.call_id / litellm.provider.model, hidden_params, all 33 metadata.* keys. Session answer unchanged (no session key in either dialect). Demonstrated in captures/litellm_semconv_probe (3 spans named `chat openai/gpt-4o-mini`, kind=3, no raw child), which is not a golden capture. That capture was recorded 2026-08-06, BEFORE the 2026-08-11 fidelity pass removed the fixture's own tracer, so it still carries 4 retired harness spans (chat_session semconv_probe / invoke_agent assistant x2 / execute_tool get_weather) in scope trace-capture.litellm, and its 3 native spans are nested under them. Only its attribute/name/kind evidence is used here — never its topology.", + "exercised_in_captures": false + }, + { + "name": "otel-v2", + "trigger": "LITELLM_OTEL_V2=true (litellm/integrations/otel/model/config.py:19, is_otel_v2_enabled()). Applies to the plain SDK too, not only the proxy: the `otel` callback constructor swaps in OpenTelemetryV2 when the flag is set (litellm_core_utils/litellm_logging.py:3834-3842). v1 remains the default.", + "effects": "A complete rewrite (litellm/integrations/otel/**). scope.name stays `litellm` (otel/logger.py:69) but the scope gains a VERSION — the litellm release string (otel/plumbing/providers.py:433-437) — where v1 emits none, so scope.version is the cheapest v1/v2 discriminator. Span vocabulary changes wholesale (otel/model/spans.py:177-206): `{operation} {model}`, `execute_guardrail {name}`, `{method} {route}`, `tools/call {tool}`, `tools/list`, `{service} {call_type}`. Attributes move into the litellm.* namespace — litellm.metadata.* replaces bare metadata.*, litellm.cost.* / litellm.guardrail.* / litellm.api_key.hash / litellm.end_user.id appear, and bare `hidden_params` disappears (otel/model/semconv.py) — so key_prefix(litellm.) is the one matcher in this seed that survives v1→v2 unchanged. gen_ai.provider.name replaces gen_ai.system. Span status becomes UNSET instead of the explicit OK. Session verdict is UNCHANGED: otel/model/semconv.py:81 declares CONVERSATION_ID = gen_ai.conversation.id, but the constant has ZERO call sites in 1.95.0 — v2 emits no session key either. New risk: v2's mapper_names (otel/model/config.py:154-162) can emit openinference / langfuse / langtrace / weave vocabularies onto the SAME spans, and LITELLM_OTEL_LEGACY_COMPAT defaults to TRUE (config.py:140), appending Traceloop-named duplicates of every key.", + "exercised_in_captures": false + }, + { + "name": "no-primary-span", + "trigger": "USE_OTEL_LITELLM_REQUEST_SPAN unset/false AND a parent span is visible (metadata['litellm_parent_otel_span'], a proxy span, a traceparent header, or the active span in the global context). This is the DEFAULT for every application that wraps completion()/acompletion() in its own span — i.e. for every framework that uses LiteLLM as a passthrough. opentelemetry.py:1206 and 1936: should_create_primary_span = parent_span is None or get_secret_bool('USE_OTEL_LITELLM_REQUEST_SPAN').", + "effects": "LiteLLM creates NO litellm_request span. Instead it calls set_attributes(parent_span, …) (opentelemetry.py:1222-1226) — every gen_ai.*, llm.*, litellm.*, metadata.* and hidden_params key is stamped onto the CALLER's span, which belongs to another vendor's scope. Consequences for this file: the whole litellm_request population vanishes; key_prefix(litellm.) starts matching foreign-vendor spans and mis-attributes them to litellm; the scope matcher no longer sees any LLM payload; only raw_gen_ai_request (re-parented to the caller's span, opentelemetry.py:1229) still arrives in the litellm scope. Worse, LiteLLM's success callback fires AFTER a synchronous caller's `with` block has exited, so the parent span is already ended and the OTel SDK drops every set_attribute with 'Setting attribute on ended span' — a silent total loss of LLM telemetry with no error surfaced anywhere (this is exactly what the first run of this fixture hit; see NOTES.md quirk #1). WIRE-VERIFIED COMPLEMENT (2026-08-11): the golden captures now exercise the OTHER branch of the same line. With no ambient parent span anywhere in the process — no wrapper span, no litellm_parent_otel_span, no traceparent — _get_span_context returns (None, None) at priority 4 (opentelemetry.py:2801-2803) and should_create_primary_span is True by the `parent_span is None` clause, WITHOUT the env override. The observed result is the customer default shape: every acompletion() produces its OWN independent trace whose ROOT is litellm_request (17 traces, 17 roots, 0 with a foreign parent), with raw_gen_ai_request as its only child — 2 spans per trace, exactly. So both halves of the branch are now documented: no parent => root-of-its-own-trace primary span (wire), parent => no span at all (source). Nothing in between exists.", + "exercised_in_captures": false + }, + { + "name": "primary-span-forced", + "trigger": "USE_OTEL_LITELLM_REQUEST_SPAN=true (get_secret_bool, opentelemetry.py:1206 and 1936) WHILE a parent span is visible. Not a default: it is the documented opt-out an integrator sets to stop LiteLLM stamping its attributes onto a foreign span. This is what the litellm captures were recorded under before 2026-08-11.", + "effects": "Restores the litellm_request span even under a parent, so the LLM payload lands on a LiteLLM-scoped span instead of the caller's — the classification story of this file becomes the passthrough-safe one. TOPOLOGY changes fundamentally and this is the field to watch: litellm_request is no longer a root, the per-request traces collapse into the caller's single trace, and root_arrival/first_batch stop being vacuous (LiteLLM builds its span retroactively inside the success callback, so it always ENDS before its parent — root-last, multi-batch). Attribute vocabulary, scope, session verdict and every matcher in this file are unaffected.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "llamaindex", + "seed": "frameworks/llamaindex/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "resource", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "service.name", + "value": "llamaindex.opentelemetry" + }, + "source": "source_code", + "justification": "LlamaIndexOpenTelemetry's DEFAULT resource, hardcoded at base.py:363-366 (Resource(attributes={SERVICE_NAME: 'llamaindex.opentelemetry'})) and used verbatim whenever the app passes neither service_name_or_resource= nor tracer_provider=. It is a library-chosen VALUE in an app-owned key, so it is a weak fingerprint only: any app that names its service is invisible to it, and it cannot be sufficient because a resource is process-wide and shared with every other instrumentation in the process. NOT observed on the wire — both captures take the app-owned-resource variant (service.name=llamaindex- [H]).", + "priority": 29960 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "llamaindex.opentelemetry.tracer" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded in llama_index/observability/otel/base.py:398 (trace.get_tracer(\"llamaindex.opentelemetry.tracer\")) — library-owned, namespaced, and the ONLY scope in either capture (79/79 + 75/75 spans). It is the only rule that classifies the 11 attribute-less `*.run` spans, which are also the trace roots. get_tracer() is called with no version and no schema_url, so scope.version and scope.schema_url are empty strings on every span and cannot be used to distinguish LlamaIndex releases.", + "priority": 39987 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "llamaindex." + }, + "source": "wire", + "priority": 29959 + } + ], + "session_candidates": [ + { + "key": "llamaindex.run_id", + "authority_predicate": null, + "validation": [ + "non_empty" + ], + "granularity": "run", + "verdict": "A" + } + ], + "decoy_keys": [ + { + "key": "session_id", + "source": "source_code", + "why": "llama_index.core.memory.Memory.session_id (memory.py:254, defaulting to generate_chat_store_key()) is the SQL/chat-store partition key that the memory blocks read and write. It is the one thing in LlamaIndex actually named a session — and it NEVER reaches telemetry: no span attribute, no event attribute, 0 occurrences in either capture. Joining on it is impossible, and expecting it is the natural mistake." + }, + { + "key": "span_id", + "source": "wire", + "why": "Present on every dispatcher span EVENT (36+9+6 events per capture). It is LlamaIndex's own string span id, format `ClassName.method-` — NOT the OTel span id and not stable across anything. Session-shaped (a uuid4) and sitting next to tags.llamaindex.run_id in the same event attribute map." + }, + { + "key": "id_", + "source": "wire", + "why": "Per-EVENT uuid4 on every span event; changes several times per span. The most session-shaped bare uuid in the capture." + }, + { + "key": "llamaindex.step.input_event", + "source": "wire", + "why": "The workflow event CLASS NAME that triggered the step (AgentSetup, ToolCall, AgentWorkflowStartEvent, WeatherTask, ...): 73/79 and 69/75 spans, only 6 distinct values, constant across runs. A low-cardinality string that a naive group-by could mistake for a conversation/thread key." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [ + { + "key": "llamaindex.agent_name" + }, + { + "key": "llamaindex.agent_model" + } + ], + "caveats": [ + { + "id": "payload_lives_in_span_events", + "text": "The defining property of this dialect: the classification-relevant surface (attributes) and the value-relevant surface (span events) are disjoint. Three attribute keys total; everything a consumer wants — prompts, tool arguments, tool results, agent replies, model name — is in flatten_dict()ed span EVENT attributes (LLMChatStartEvent.messages, model_dict.model_name, step.output.output, workflow.output.output). Any ingest path that drops span events keeps 100% of the classification signal and loses ~82% of the payload." + }, + { + "id": "no_model_no_tokens_no_cost", + "text": "There is no model name, no token count and no cost on any span, in any signal. Model identity exists only inside the LLMChatStartEvent event (model_dict.model_name), which is present on 9/18 and 8/16 astream_chat spans. Token usage exists NOWHERE: LLMChatEndEvent/LLMChatInProgressEvent are dropped because the streaming span closes when the coroutine returns the generator, before the stream is consumed (base.py:306 `if current_span_id not in self.span_handler.all_spans`). Per-span cost/token rollups are impossible for LlamaIndex." + }, + { + "id": "double_llm_spans", + "text": "Every LLM call emits TWO OpenRouter.astream_chat spans with near-identical timing (FunctionCallingLLM.astream_chat_with_tools -> astream_chat, both dispatcher-decorated): 18 spans / 9 LLMChatStartEvents in llamaindex_user, 16/8 in llamaindex_agents. Only the first carries the event. Any LLM-call count keyed on span name double-counts; dedupe on (parent, LLMChatStartEvent presence)." + }, + { + "id": "hitl_error_spans_are_not_failures", + "text": "NATIVE HITL: ctx.wait_for_event(HumanResponseEvent, waiter_event= InputRequiredEvent(...)) suspends by raising an internal control-flow exception and REPLAYING the step, so each approval produces two FunctionTool.acall spans — the first with StatusCode.ERROR, a SpanDropEvent and an `exception` event whose message is `Waiting for event `, then a successful one. 2 of the 3 ERROR spans in the corpus-for-this-framework are this pattern, not failures. There is no HITL-specific attribute or event name: the approval gate is only detectable by that exception message string, which the algebra cannot match." + }, + { + "id": "error_status_is_native_and_shallow", + "text": "Unlike google_adk, the ERROR status and the `exception` span event are NATIVE here (base.py:245-255 prepare_to_drop_span calls record_exception + set_status). The real tool failure (fetch_transport_data 503) is marked ERROR on FunctionTool.acall only; the parent BaseWorkflowAgent.call_tool stays OK because the agent converts the exception into a ToolCallResult. exception.stacktrace frames are dispatcher internals, not customer code." + }, + { + "id": "instrument_tags_is_the_only_extension_point", + "text": "[H] scenario_b.py uses instrument_tags({'agent_name','agent_model'}) around each worker run, producing llamaindex.agent_name/llamaindex.agent_model on 62/75 spans. Those keys are FIXTURE-ONLY and must not be transcribed. They are kept visible because they prove two registry-relevant facts: (a) LlamaIndex has NO native agent-identity attribute — without tags, agent identity is recoverable only from prompt text inside span events; (b) instrument_tags is the injection point for any customer key, session ids included, and it prefixes dot-less keys with 'llamaindex.' while passing dotted keys through verbatim (base.py:191-196). NOT counted under harness_mutated_spans (fixed after cross-seed audit): passing config through a first-class API (instrument_tags) is simulated customer action, exactly like mastra's tracingOptions.metadata — 'mutated' is reserved for scenario code calling telemetry APIs on native spans (record_exception/set_status/ add_event, the google_adk case). The injected VALUES stay [H] and the keys are listed under harness_keys." + }, + { + "id": "instrument_tags_injection", + "text": "Because the only injection point is a free-form tag dict, there is NO canonical LlamaIndex session attribute key, and the seed carries no session candidate (the earlier 'session.id' convention-bet candidate was removed by the cross-seed audit — unobserved, non-native keys are not registrable). The mechanism itself, for the education surface: wrap each run in `with instrument_tags({'': sid}): await agent.run(...)` — tags are read at span creation (dispatcher.py:361-375), so the key lands on the ROOT and every descendant (unlike run_id, applied post-root); a dotted key is written verbatim, a dot-less key becomes llamaindex. (base.py:191-196). Session support therefore requires per-tenant configuration (a per-org overlay rule or the key_prefix-shaped candidate recorded in algebra_violations)." + }, + { + "id": "thinnest_resource_in_the_corpus", + "text": "The resource has TWO attributes (service.name plus the fixture's framework=llamaindex [H]) and NO telemetry.sdk.* at all — not language, not name, not version. This is not only a fixture artefact: the library's own default builds the resource with the bare Resource(attributes=...) constructor rather than Resource.create() (base.py:363-366,382), so the default customer path also ships a resource with a single attribute and no SDK identification. Any ingest heuristic keyed on telemetry.sdk.language sees nothing here." + }, + { + "id": "run_id_is_the_only_native_correlator", + "text": "llamaindex.run_id is a genuinely useful correlator at RUN granularity (it groups the 5 agent sub-runs inside the llamaindex_agents trace and survives across the batch boundary), but it is one level below session and one level below trace in llamaindex_agents. Recording it as a candidate at granularity: run is what keeps state-5 visible in key_state_by_candidate; the max-reduction would otherwise print the same histogram as a framework with no keys at all." + } + ], + "variants": [ + { + "name": "app-owned-resource", + "trigger": "The app passes tracer_provider= (or service_name_or_resource=) to LlamaIndexOpenTelemetry. Default path (neither passed) builds TracerProvider(resource=Resource(attributes={service.name: 'llamaindex.opentelemetry'})) — base.py:363-366,381-382. The fixture takes the app-owned path, so BOTH captures show the variant, not the default.", + "effects": "Only the resource block changes. Default => the sole resource attribute is service.name='llamaindex.opentelemetry', which resource_matchers can fire on. App-owned => service.name is the app's, and the resource carries telemetry.sdk.* only if the app built it with Resource.create() (the fixture used the bare Resource() constructor, hence a 2-key resource with NO telemetry.sdk.*). No span, scope, attribute, event or session field changes.", + "exercised_in_captures": true + } + ] + }, + { + "vendor": "mastra", + "seed": "frameworks/mastra/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "resource", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "telemetry.sdk.name", + "value": "@mastra/otel-exporter" + }, + "source": "wire", + "justification": "The only NATIVE framework-identifying resource attribute in the corpus. Mastra abuses the standard OTel key: SpanConverter.initIfNeeded() (otel-exporter dist/index.js:585-600) builds the resource from scratch with telemetry.sdk.name = the exporter package name and telemetry.sdk.version = the exporter version (1.3.8), while service.version carries the @mastra/core version. SUFFICIENT is justified by construction, not by single-vendor-process assumption: this resource is minted per exported span inside Mastra's own converter — no resource detector, no merge with a global SDK Resource — and Mastra never routes spans through a process-wide TracerProvider. A Mastra app also running @opentelemetry/auto-instrumentations-node therefore emits its HTTP / DB spans with the NodeSDK's own resource (telemetry.sdk.name=opentelemetry); those spans cannot inherit this value. The only ways to see it on a non-Mastra span are a customer explicitly setting OTEL_RESOURCE_ATTRIBUTES or the exporter's own resourceAttributes option to this value (it is merged last and would win), or a collector rewriting resources. Both are pathological; the scope matcher agrees on 103/103 spans in these captures, so a wrong sufficient verdict changes nothing here. NOTE the asymmetry: the LOG signal's resource is built separately (dist/index.js:936-941) and carries service.name ONLY, so this predicate classifies spans but never log records.", + "priority": 39986 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "@mastra/otel-exporter" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded: SpanConverter sets scope = {name: params.packageName, version: } with packageName pinned to the literal '@mastra/otel-exporter' at construction (otel-exporter dist/index.js:602-605, 910). Library-owned, npm-namespaced, stable across the package; catches 66/66 + 37/37 spans and the single log record (LoggerProvider.getLogger uses the same name, dist/index.js:944). Nothing else can land in this scope: Mastra converts finished AISpans into ReadableSpans and pushes them straight into its own BatchSpanProcessor (dist/index.js:963-967), so no third-party instrumentation can write to it. The scope identifies the EXPORTER, not the framework version — scope.version tracks @mastra/otel-exporter, and @mastra/core's version is only visible as service.version.", + "priority": 39985 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "mastra." + }, + "source": "wire", + "priority": 29958 + } + ], + "session_candidates": [ + { + "key": "gen_ai.conversation.id", + "authority_predicate": { + "op": "present", + "key": "mastra.span.type" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "B" + }, + { + "key": "mastra.metadata.runId", + "authority_predicate": { + "op": "present", + "key": "mastra.span.type" + }, + "validation": [ + "non_empty" + ], + "granularity": "run", + "verdict": "A" + }, + { + "key": "mastra.metadata.resourceId", + "authority_predicate": { + "op": "present", + "key": "mastra.span.type" + }, + "validation": [ + "non_empty" + ], + "granularity": "user", + "verdict": "B" + } + ], + "decoy_keys": [ + { + "key": "mastra.metadata.resumedFromSpanId", + "source": "wire", + "why": "Session-shaped hex id on all 15 resume-population spans in mastra_user, and it looks like a correlation key because it equals the resumed root's dangling parentSpanId. It is a SPAN id (of an internal, never-exported span), scoped to one suspend→resume hop; two different values inside one 66-span session." + }, + { + "key": "mastra.metadata.resumed", + "source": "wire", + "why": "Constant literal 'true' on the 15 resume spans. A boolean marker, not an identifier — but it is the only span-local way to recognise the resume roots whose parents dangle (see algebra_violations)." + }, + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "On every `chat` span with a response (7/66 + 5/37); one distinct value per LLM call, 7 values inside the single mastra_user session." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id (call_… / toolu_bdrk_…). Per tool invocation; the one value that repeats (call_nGuNvnEH… x2 in mastra_agents) does so only because that span was exported twice." + }, + { + "key": "service.name", + "source": "wire", + "why": "mastra- here [H]; in real deployments it is the Observability config's serviceName — process/deployment identity, never session identity. Note it is also the ONLY attribute on the log signal's resource." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [ + { + "key": "mastra.metadata.scenario" + }, + { + "key": "mastra.metadata.capture" + }, + { + "key": "mastra.metadata.turn" + }, + { + "key": "mastra.metadata.approval" + } + ], + "caveats": [ + { + "id": "dangling_parents_on_resume", + "text": "Mastra's HITL resume re-parents the new root onto the span that was live when the run suspended (@mastra/core dist/workflow-event-processor-BbED1LMn.js:2207-2215 persists {traceId, spanId}; dist/agent-Dj30gJa3.js:34499-34506 feeds it back as tracingOptions.parentSpanId). Under the default includeInternalSpans: false that span is an INTERNAL loop span that is never exported, and BaseSpan.getParentSpanId returns the persisted id verbatim without the usual internal-span skip (@mastra/observability dist/index.js:2579-2584). Result: 2 spans in mastra_user declare parents 85b7af4294e4b0ef / 5179d6fcaf30c7ba that no record ever contains. Root detection MUST be 'parent absent from the trace' — mastra_user then has 9 roots across 7 traces, with 2 traces holding 2 roots each." + }, + { + "id": "duplicate_span_id_with_conflicting_error", + "text": "mastra_agents record seq 0 contains `execute_tool fetch_transport_data` TWICE under one span id (9c1bf272f056c34a) — same trace, same parent, same status (ERROR/503), same exception event, but error.type=unknown on the first copy and error.type=TOOL_EXECUTION_FAILED on the second. error.type is `span.errorInfo.id || 'unknown'` (otel-exporter dist/index.js:487), so the tool span was ended twice with progressively-enriched errorInfo and both ends were exported. 37 spans, 36 distinct span ids. See algebra_violations." + }, + { + "id": "no_gen_ai_system", + "text": "Mastra emits NO gen_ai.system. Provider attribution lives in gen_ai.provider.name, normalized from the model string by an alias table (dist/index.js:498-557): every call here went to OpenRouter, so the value is the literal 'openrouter' (not in the semconv's provider list) while gen_ai.request.model carries openai/gpt-4o-mini or anthropic/claude-haiku-4.5. Any rule keyed on gen_ai.system misses Mastra entirely." + }, + { + "id": "operation_name_is_not_semconv", + "text": "gen_ai.operation.name doubles as Mastra's span-type field: 6 of the 8 observed values (model_step, model_inference, memory_operation, processor_run, workflow_step, workflow_parallel) are not semconv operation names, they are span.type.toLowerCase(). Convenient for coverage, hostile to any consumer that validates the enum." + }, + { + "id": "nested_model_spans_double_count", + "text": "One LLM call produces up to three nested spans: `chat ` (CLIENT, the only one with gen_ai.request/response/usage), `model_step ` and `model_inference ` (both INTERNAL, carrying mastra.model_step.input/output blobs). In mastra_user that is 9 chat / 9 model_step / 9 model_inference for 9 LLM calls; in mastra_agents 5 / 8 / 8 (multi-step tool loops). Token and cost rollups must count `chat` spans only." + }, + { + "id": "span_kind_is_almost_useless", + "text": "getSpanKind maps MODEL_GENERATION / RAG_EMBEDDING / MCP_TOOL_CALL to CLIENT and everything else to INTERNAL (dist/index.js:657-664). So 9+5 CLIENT spans and 89 INTERNAL — the trace root is INTERNAL even in the workflow scenario." + }, + { + "id": "attribute_typing_is_mixed", + "text": "Unlike the Java frameworks, values keep their OTLP types: token counts arrive as intValue, temperature as doubleValue, gen_ai.response.finish_reasons as a JSON STRING '[\"stop\"]' (JSON.stringify of a one-element array, dist/index.js:449) and gen_ai.tool.definitions as a JSON string array. Canonical stringification is required before any eq() comparison." + }, + { + "id": "harness_metadata_shares_the_framework_namespace", + "text": "[H] The scenarios pass tracingOptions.metadata {scenario, turn, capture, approval}; the exporter writes them into mastra.metadata.* alongside the framework's own runId/threadId/resourceId. Zero spans are harness-EMITTED and zero are harness-MUTATED (no manual span API is used anywhere under frameworks/mastra/ — grep for startActiveSpan/setAttribute/recordException/addEvent returns nothing), but 103/103 spans carry harness-authored attribute keys that a customer's own metadata would occupy identically. See session.harness_keys." + }, + { + "id": "exports_nothing_without_an_explicit_exporter", + "text": "Mastra reads no OTEL_EXPORTER_OTLP_* variable: OtelExporter requires an explicit provider config and disables itself with '[OtelExporter] Custom configuration requires endpoint' otherwise, and `observability` must be an Observability INSTANCE — passing a plain config object silently installs NoOpObservability. Both failure modes are silent, which bounds how much Mastra data arrives from apps that were merely env-configured." + }, + { + "id": "single_resource_block_per_record", + "text": "Every record is one resourceSpans block with one scopeSpans block containing the whole batch (18/29/19 spans for mastra_user), so the resource and scope are NOT duplicated per span — the opposite of spring_ai's one-span-per-block shape." + } + ], + "variants": [ + { + "name": "include-internal-spans", + "trigger": "Observability config includeInternalSpans: true (default false — @mastra/observability dist/index.js:2903). Fixture knob INCLUDE_INTERNAL_SPANS=1.", + "effects": "Mastra models the agent loop as nested workflows, so the internal tier is most of the telemetry: the pre-prune mastra_user run recorded 331 spans / 5.5 MB of OTLP body for the same 6 turns that produce 66 spans / 178 KB with it off (record seq1 alone was 4,216,873 B on the wire, the largest single record in the corpus — still in records.jsonl.bak-2026-08-06T02-20-40-368Z). It changes topology, span populations (invoke_workflow -input/output-processor, workflow_step assistant), root_arrival and largest_record_bytes; the *-processor workflows surface as small parentless traces of their own. It also REPAIRS the dangling parents of resume roots (see caveats), because the persisted suspend span id is an internal span that the default config never exports. No attribute key, scope, resource or session key changes.", + "exercised_in_captures": false + }, + { + "name": "exclude-span-types", + "trigger": "Observability config excludeSpanTypes: [...] (default: no exclusions). The fixture sets excludeSpanTypes: [SpanType.MODEL_CHUNK].", + "effects": "Silently drops whole span types before the exporter sees them. Default Mastra emits one MODEL_CHUNK span per streamed token, so an unmodified streaming customer app ships one span per token that these captures do not contain — a volume difference of orders of magnitude, not a schema difference. No key, scope or session-key change; classification rules are unaffected.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "microsoft_agent_framework", + "seed": "frameworks/microsoft_agent_framework/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "agent_framework" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded as the default instrumenting_module_name in agent_framework/observability.py:992-996 (get_tracer), with instrumenting_library_version = the package version. ONE scope for the whole framework — agent, chat, tool, workflow, executor, edge-group and message spans all land in it (53/53 spans across both captures, no schemaUrl, no scope attributes). It must be sufficient: the 17 `chat ` spans carry no framework-specific attribute at all and nothing else in this seed classifies them. Accepted cost: get_tracer() and create_workflow_span() are PUBLIC API, so app-authored spans land in this scope too — see false_positives.", + "priority": 39984 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.provider.name", + "value": "microsoft.agent_framework" + }, + "source": "wire", + "priority": 29957 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "agent_framework." + }, + "source": "wire", + "priority": 29956 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "executor." + }, + "source": "wire", + "priority": 29955 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "edge_group." + }, + "source": "wire", + "priority": 29954 + } + ], + "session_candidates": [ + { + "key": "gen_ai.conversation.id", + "authority_predicate": { + "op": "eq", + "key": "gen_ai.operation.name", + "value": "invoke_agent" + }, + "validation": [ + "non_empty", + "not_in_decoy_values" + ], + "granularity": "session", + "verdict": "B" + } + ], + "decoy_keys": [ + { + "key": "gen_ai.agent.id", + "source": "wire", + "why": "THE trap in this framework. Per-Agent-OBJECT uuid4 (_agents.py:436-438), on all 13 invoke_agent spans. In the user capture ONE value covers all 8 traces and it is the only cross-trace constant that exists — a session-shaped coincidence of one process holding one Agent object. The agents capture shows the truth: 5 distinct values inside a single workflow run. In a server the Agent is a module-level singleton, so joining on it merges every user of the process into one 'session'. Customers may also pass Agent(id=...) explicitly, which makes it stable across restarts and even more convincing. It is agent identity, never conversation identity." + }, + { + "key": "workflow.id", + "source": "wire", + "why": "Per WorkflowBuilder.build() uuid4. Genuinely joins the two agents-capture traces (identical value on workflow.build and workflow.run), which makes it look like a session key — but a server builds the workflow once at import and runs it for every request, so the value is process-constant across unrelated runs. Use it to stitch build->run, never to define a session." + }, + { + "key": "service.instance.id", + "source": "wire", + "why": "uuid4 minted per process by the OTel SDK resource detector; one value per capture, so it is perfectly session-shaped in a one-process fixture and perfectly useless in a server. Process identity." + }, + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "Provider response id on every `chat` span; unique per LLM call." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id (call_... / toolu_...). It repeats across the HITL approval span and the resumed execute_tool span, which makes it look like a correlation key for the approval flow; it correlates one tool call, not a session." + }, + { + "key": "edge_group.id", + "source": "wire", + "why": "'/' per edge group; the 3 FanIn spans share one value and look like a group id. Scoped to the workflow graph, not to a conversation." + } + ], + "decoy_values": [ + { + "value": "agent_framework_local_history_persistence", + "source": "source_code", + "why": "LOCAL_HISTORY_CONVERSATION_ID sentinel (_sessions.py:1028). Stamped onto ChatResponse.conversation_id when require_per_service_call_history_persistence is used without service-side storage (_sessions.py:1341-1343). The framework itself guards against writing it into AgentSession.service_session_id (is_local_history_conversation_id checks at _agents.py:1135, 1170, 1193), but an integration that feeds response.conversation_id back in as the next run's conversation_id option would stamp it on every invoke_agent span as a process-wide constant. Not observed on the wire." + }, + { + "value": "unknown", + "source": "source_code", + "why": "Fallback used across the telemetry layer when an identity cannot be resolved (gen_ai.agent.id/gen_ai.agent.name default to 'unknown' at observability.py:1833-1834, provider name at 1421). Never a session id; listed so a resolver never treats a literal 'unknown' as a real value." + } + ], + "harness_keys_fixture_only": [ + { + "key": "hitl.approval.id" + }, + { + "key": "hitl.mechanism" + }, + { + "key": "hitl.tool.name" + }, + { + "key": "hitl.tool.arguments" + }, + { + "key": "hitl.user.response" + }, + { + "key": "hitl.approved" + } + ], + "caveats": [ + { + "id": "otlp_protocol_defaults_to_grpc", + "text": "_get_exporters_from_env() reads OTEL_EXPORTER_OTLP_PROTOCOL with default=\"grpc\" (observability.py:554), against the OTel spec default of http/protobuf. A customer pointing OTEL_EXPORTER_OTLP_ENDPOINT at an HTTP collector and setting nothing else sends gRPC to an HTTP port and captures NOTHING — silently, with retries, no error surfaced to the app. Highest-frequency onboarding failure to expect for this vendor. The fixture sets the var explicitly (telemetry.py:25)." + }, + { + "id": "cumulative_histograms_reexported_in_full", + "text": "All three instruments are CUMULATIVE (aggregationTemporality=2) behind a PeriodicExportingMetricReader at 5 s (observability.py:983). Every export re-ships every data point ever recorded: in the user capture seq 5 and seq 6 are byte-identical metric payloads, as are seq 6 and seq 7 in the agents capture. Any naive sum over records double-counts, and metric volume grows with process lifetime, not with traffic." + }, + { + "id": "unbounded_metric_cardinality_on_tool_duration", + "text": "agent_framework.function.invocation.duration is recorded with the execute_tool SPAN's attribute dict (_tools.py:793), so its dimensions include gen_ai.tool.call.id — a unique id per invocation — plus gen_ai.tool.description and, under ENABLE_SENSITIVE_DATA, the full gen_ai.tool.call.arguments JSON. Every tool call therefore creates a NEW permanent cumulative time series carrying prompt-derived content. Combined with cumulative temporality this is an unbounded, monotonically growing metric payload; wire-visible already at 3 series in a 32-span capture." + }, + { + "id": "metric_views_drop_everything_else", + "text": "configure_otel_providers() installs create_metric_views(): View(instrument_name=\"agent_framework*\"), View(instrument_name=\"gen_ai*\"), View(instrument_name=\"*\", aggregation=DropAggregation()) (observability.py:688-692) on the GLOBAL MeterProvider. Calling the framework's convenience setup therefore silently discards every other library's metrics in the process — HTTP clients, runtime metrics, the app's own counters." + }, + { + "id": "framework_overwrites_service_version", + "text": "create_resource() sets service.version to the agent-framework package version when OTEL_SERVICE_VERSION is unset (observability.py:669-671), and service.name to the literal 'agent_framework' when OTEL_SERVICE_NAME is unset. Both captures show service.version=1.13.0 for an app that has no such version. Resource-based app identity from this framework is not trustworthy." + }, + { + "id": "gen_ai_provider_name_is_the_client_class", + "text": "gen_ai.provider.name is a ClassVar of whichever client/agent class emitted the span, not the endpoint actually contacted: 'microsoft.agent_framework' on invoke_agent (_agents.py:754), 'openai' on chat (agent_framework_openai _chat_completion_client.py:1193) — for traffic that went to OpenRouter, including the anthropic/claude-haiku-4.5 calls. server.address carries the truth (https://openrouter.ai/api/v1/ on 17/17 chat spans; the 'Unknown' first-request value NOTES.md reported does not occur in these captures)." + }, + { + "id": "span_links_are_the_only_fan_in_encoding", + "text": "Workflow message delivery is modelled with OTel span links, not parent/child: every executor.process / edge_group.process span is a direct child of workflow.run and links back to the message.send span that published its input. 8 spans carry links in the agents capture; executor.process summary_agent carries 3 — the fan-in join. A parser walking only parentSpanId sees flat siblings and loses the causality entirely. Links are outside the predicate algebra (see algebra_violations)." + }, + { + "id": "no_native_hitl_telemetry", + "text": "@tool(approval_mode=\"always_require\") is a genuine native gate — the loop stops before the tool body runs — but it emits nothing: no span, no event, no attribute. A DENIED call produces no execute_tool span at all, so the only wire evidence is a string inside the next chat span's gen_ai.input.messages. Approval latency and approval outcome are unobservable for this vendor." + }, + { + "id": "harness_reparents_the_resume_traces", + "text": "[H] The 2 `approval delete_file` spans (scenario_a_user.py:58, via the public create_workflow_span) WRAP the resuming agent.run(), so in traces 17bcfafb and e12727cd the harness span is the ROOT and the native invoke_agent is its child. Without the fixture those two resume runs would be ordinary invoke_agent-rooted traces; the trace COUNT (8) is unaffected, the root identity is not. Root-anchored readers must not learn 'MAF traces are rooted at a non-gen_ai span' from this capture." + }, + { + "id": "attribute_value_typing_is_mixed", + "text": "Unlike the Java frameworks, values arrive natively typed: gen_ai.usage.input_tokens as intValue, gen_ai.request.temperature and agent_framework.function.invocation.duration as doubleValue, gen_ai.response.finish_reasons as an arrayValue of strings. Canonical stringification (verify-seed.ts semantics) is what makes eq() predicates portable here; do not assume stringValue." + }, + { + "id": "semconv_flavour_is_latest_experimental_without_opt_in", + "text": "1.13.0 emits the gen_ai_latest_experimental flavour unconditionally — message content as gen_ai.input.messages / gen_ai.output.messages JSON attributes, no gen_ai.user.message / gen_ai.choice SPAN events, gen_ai.provider.name instead of gen_ai.system, only the new gen_ai.usage.input_tokens/output_tokens names. There is no OTEL_SEMCONV_STABILITY_OPT_IN switch anywhere in the package (grep: zero hits), so there is no stable-flavour variant to model — but any consumer keyed on gen_ai.system or on the older event names sees nothing from this vendor." + } + ], + "variants": [ + { + "name": "sensitive-data-off", + "trigger": "ENABLE_SENSITIVE_DATA unset or false — THIS IS THE DEFAULT (observability.py:839 SENSITIVE_DATA_ENABLED). Both captures were recorded with it forced to true, so the captures show the NON-default payload shape.", + "effects": "Removes gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions from chat/invoke_agent spans and gen_ai.tool.call.arguments / gen_ai.tool.call.result from execute_tool spans and from the agent_framework.function.invocation.duration metric's attribute set. NOT gated and therefore unaffected: gen_ai.tool.definitions (emitted through OTEL_ATTR_MAP at observability.py:2561-2566, no sensitive-data check — NOTES.md previously claimed otherwise). No scope_matcher, attr_matcher, session candidate or granularity answer in this file changes; only payload size and the metric cardinality caveat do.", + "exercised_in_captures": false + }, + { + "name": "framework-logger-at-info", + "trigger": "logging.getLogger(\"agent_framework\").setLevel(logging.INFO) (or a root basicConfig at INFO) together with configure_otel_providers(), which attaches an OTLP LoggingHandler to exactly that logger (observability.py:968-975). The logger inherits WARNING by default, which is why both captures carry only ERROR/WARN records.", + "effects": "Every prompt and completion is duplicated into the LOG signal: _capture_messages() calls logger.info(otel_message, extra={\"event.name\": gen_ai.system.message | gen_ai.user.message | gen_ai.assistant.message | gen_ai.tool.message | gen_ai.choice, \"gen_ai.provider.name\": ...}) (observability.py:2707-2723). Flips signals.logs.ai_data reasoning and carries_data_spans_lack; the names land as a log ATTRIBUTE event.name, never in the OTLP eventName field (the SDK LoggingHandler does not promote extras to event_name), so signals.logs.event_names_observed stays empty either way.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "openai_agents_sdk", + "seed": "frameworks/openai_agents_sdk/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.openai_agents" + }, + "owned_by": "library", + "source": "wire", + "justification": "Library-owned and hardcoded: the instrumentor calls trace_api.get_tracer(__name__, __version__, tracer_provider) where __name__ is the package path openinference.instrumentation.openai_agents (openai_agents/__init__.py:41-43); scope.version is the INSTRUMENTOR release (1.6.2), never the SDK release. It is the only scope in either capture and catches 44/44 + 32/32 spans, including the trace-root span, which carries a single attribute and is otherwise unclassifiable. Sufficient because the OITracer built here is used by nothing else in the process: every span in this scope originates in the SDK's own tracing pipeline (agents.tracing) or the realtime wrappers.", + "priority": 39983 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "op": "present", + "key": "openinference.span.kind" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "C", + "source": "source_code" + }, + { + "key": "gen_ai.conversation.id", + "authority_predicate": { + "op": "present", + "key": "openinference.span.kind" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "C", + "source": "source_code" + } + ], + "decoy_keys": [ + { + "key": "service.instance.id", + "source": "wire", + "why": "Stock OTel-Python resource attribute, a fresh uuid4 per PROCESS (6a9de051-... / b944a5f5-...). In these fixtures one process is one scenario, so it looks exactly like a session id and is constant across all 6 traces of openai_agents_sdk_user. In a server it is one value for millions of conversations." + }, + { + "key": "graph.node.id", + "source": "wire", + "why": "13/76 spans (8 + 5). Despite the name it is the AGENT NAME copied verbatim (_processor.py:183, from AgentSpanData.name) — `assistant`, `budget_worker`. A constant per agent definition, not an identifier of anything runtime." + }, + { + "key": "llm.output_messages.0.message.tool_calls.0.tool_call.id", + "source": "wire", + "why": "Provider tool-call id (call_...), and its mirror llm.input_messages.N.message.tool_call_id. Repeats across the approval-pending and executed delete_file spans of one HITL turn, which makes it look like a correlation key; it correlates one tool call, never a session." + }, + { + "key": "openinference.project.name", + "source": "source_code", + "why": "Resource attribute set by openinference.instrumentation.using_project (_projects.py:20-24). A Phoenix PROJECT name — a deployment-wide constant, the same decoy shape as langsmith.trace.session_name. Never a session." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "group_id_is_silently_dropped", + "text": "THE landmine. The SDK's documented session idiom — trace(workflow_name, group_id=...) and RunConfig.group_id, described in its own docs as the way to link multiple traces from one conversation — is accepted, stored on the Trace object (tracing/traces.py:168,231-232, exported to OpenAI's backend as payload['group_id']) and then thrown away by the OTLP path: OpenInferenceTracingProcessor.on_trace_start reads ONLY trace.name (_processor.py:87-94). grep group_id in the instrumentation package = 0 matches; the fixture used the idiom correctly on all 6 turns and the value scenario-a- appears 0 times in the capture bytes. A customer who did everything the SDK told them to do produces 100% unsessioned traces, with no error and no warning. Same for RunConfig.trace_metadata (Trace.metadata is never read either)." + }, + { + "id": "sdk_trace_ids_dropped", + "text": "The SDK mints its own ids — trace_<32hex> and span_<24hex> — which are the ids shown in the OpenAI traces dashboard. The bridge creates fresh W3C OTel ids and emits neither SDK id as an attribute (_processor.py has no reference to trace.trace_id outside its internal dict key). There is NO way to correlate a maple trace back to the OpenAI dashboard, in either direction." + }, + { + "id": "task_and_turn_spans_are_unmapped", + "text": "openai-agents 0.19.4 added TaskSpanData (one per Runner.run, name = the workflow name) and TurnSpanData (one per agent-loop turn, carrying turn index, agent_name and per-turn USAGE). OpenInference 1.6.2's _get_span_kind knows neither type, so both fall through to CHAIN (_processor.py:235) and on_span_end's isinstance chain matches nothing — their entire payload is discarded. Wire-confirmed: all 11 `Agent workflow` and all 17 `turn` spans carry exactly 2 attributes (openinference.span.kind + llm.system). 28/76 spans in this corpus are content-free scaffolding purely because the instrumentor lags the SDK, and the SDK's own per-run/per-turn token usage never reaches OTLP." + }, + { + "id": "trace_root_is_agent_not_chain", + "text": "The trace-root span (`user turn N`, `amsterdam_research_briefing`) is openinference.span.kind=AGENT, hardcoded at _processor.py:92 — NOT CHAIN. It is also the only span type that does not get llm.system, so it carries exactly ONE attribute. An earlier revision of NOTES.md recorded it as CHAIN; the wire disagrees (7/7 AGENT)." + }, + { + "id": "span_names_are_customer_text", + "text": "Span names are unusable as rules. The root's name is the workflow name the customer passes to trace() (free text: `user turn 3`); `Agent workflow` is merely RunConfig.workflow_name's DEFAULT; AGENT span names are agent names; TOOL span names are tool names. On a resumed run the task span inherits the workflow name from the persisted RunState (agents/run.py:607-665 resolve_trace_settings -> task_span(name= trace_workflow_name)), which is why openai_agents_sdk_user contains two spans named `user turn 6` at different depths of one trace." + }, + { + "id": "llm_system_is_always_openai", + "text": "llm.system is the constant string openai on 69/76 spans, set at span start from OpenInferenceLLMSystemValues.OPENAI (_processor.py:127) with no reference to the actual provider. openai_agents_sdk_agents routes 3 of 8 generations to anthropic/claude-haiku-4.5 through OpenRouter and still reports openai. Only llm.model_name carries the truth, and llm.invocation_parameters carries the base_url (https://openrouter.ai/api/v1/, credential-free — verified: 0 occurrences of api_key or sk-or in either capture)." + }, + { + "id": "pending_approval_repr_leak", + "text": "Native HITL (@function_tool(needs_approval=True)) fires correctly — 3 delete_file spans in openai_agents_sdk_user: one pending in the rejected turn 5, one pending plus one executed in the approved turn 6. Both PENDING spans set output.value to a 4762-byte Python repr(FunctionToolResult(...)) containing every tool's JSON schema, the nested ToolApprovalItem and live object addresses (<... object at 0x10c343740>), with output.mime_type absent; the executed span is a clean 30-byte string. Nothing in the span marks which is which — approval state is invisible to any rule." + }, + { + "id": "errors_are_status_text_only", + "text": "The corpus' only failure (fetch_transport_data, openai_agents_sdk_agents) is an OTel status code 2 whose message is the SDK SpanError rendered as a Python dict literal: \"Error running tool (non-fatal): {'tool_name': 'fetch_transport_data', 'error': 'transport data service unavailable (503)'}\". No exception event, no exception.type/message attribute, no error.type. Note the parent chain stays OK — the agent recovered — so trace-level error rollups keyed on the root see a clean trace." + }, + { + "id": "payload_is_written_twice", + "text": "Every `generation` span carries the same messages twice: flattened into indexed keys (llm.input_messages.12.message.tool_calls.1.tool_call.function.arguments) AND as a JSON blob in input.value/output.value. One 6-turn conversation with growing history reaches 8.1 KB of attributes on a single span and 55 KB in one OTLP record. There is no toggle that drops only one half." + }, + { + "id": "no_native_otel_export", + "text": "There is no OTel export in the SDK. Without openinference-instrumentation- openai-agents nothing reaches an OTLP endpoint (the SDK POSTs to api.openai.com/v1/traces/ingest instead), and calling set_tracing_disabled(True) to stop that ALSO kills the bridge — the OpenInference processor feeds off the same pipeline, so the correct wiring keeps SDK tracing enabled and replaces its processors (exclusive_processor=True, the default). With exclusive_processor=False a customer double-ships every trace to OpenAI and to maple." + }, + { + "id": "unbatched_multi_scope_shape", + "text": "OTLP shape is one resourceSpans -> one scopeSpans -> N spans per record, with the resource repeated per record. Records are BatchSpanProcessor-timed, not trace-aligned: openai_agents_sdk_user packs traces 1-3 into record 0 and traces 4-6 into record 1, while openai_agents_sdk_agents splits ONE trace across both records (23 + 9 spans) with the root arriving last. record != trace in both directions." + } + ], + "variants": [ + { + "name": "no-openinference-default", + "trigger": "openinference-instrumentation-openai-agents absent / OpenAIAgentsInstrumentor never instrumented — THE DEFAULT for a plain `pip install openai-agents` app. The SDK ships its own tracing pipeline whose default BackendSpanExporter POSTs proprietary JSON to https://api.openai.com/v1/traces/ingest (agents/tracing/processors.py); it reads no OTEL_EXPORTER_OTLP_* variable and never produces an OTel span.", + "effects": "Nothing in this file applies: zero OTLP spans reach maple. Everything below describes the OpenInference bridge, which is an explicit customer opt-in (one extra package + an instrument() call + a BYO TracerProvider). Note the proprietary payload that OpenAI receives DOES carry group_id and the SDK trace_/span_ ids that the OTLP path drops (see caveats).", + "exercised_in_captures": false + }, + { + "name": "using-session", + "trigger": "Application code wraps agent invocations in openinference.instrumentation.using_session() (or using_attributes(session_id=)). Not a framework feature and not mentioned in the OpenAI Agents SDK docs — it comes from the openinference-instrumentation base package.", + "effects": "session.id is merged into the START attributes of EVERY span this scope emits, including the trace-root span (_tracers.py:166-184 merges get_attributes_from_context() into the attributes passed to Tracer.start_span). session:candidates state flips 3 -> 6 across the board, unsessioned_traces -> 0, root_readable -> true, and sampling.key_at_root_start -> true (the sampler sees the key). This is the ONLY way an OTLP-exported openai_agents_sdk trace acquires a session key.", + "exercised_in_captures": false + }, + { + "name": "genai-semconv-dual-write", + "trigger": "OPENINFERENCE_ENABLE_GENAI_SEMCONV=true (default false — openinference/instrumentation/config.py:109,118).", + "effects": "At span END, _spans.py:75-79 dual-writes an OTel-GenAI projection of the OpenInference attributes: gen_ai.operation.name, gen_ai.provider.name, gen_ai.conversation.id (from session.id), gen_ai.request.*, gen_ai.usage.*, gen_ai.tool.*, gen_ai.input_messages/output_messages. Adds gen_ai.operation.name to fallback_fingerprints and makes the gen_ai.conversation.id candidate reachable. Written post-start, so it does NOT make head sampling possible.", + "exercised_in_captures": false + }, + { + "name": "no-task-turn-spans", + "trigger": "Runner.run(..., run_config=RunConfig(tracing={\"include_task_and_turn_spans\": False})) — default True (agents/tracing/config.py:16-18).", + "effects": "Removes the `Agent workflow` (TaskSpanData) and `turn` (TurnSpanData) spans: 15/44 spans in openai_agents_sdk_user and 13/32 in openai_agents_sdk_agents disappear and the AGENT spans become direct children of the trace root. Changes topology and every span count in the goldens; changes no classification or session answer (those spans carry only openinference.span.kind + llm.system).", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "openinference-openai", + "synthesized": true, + "justification": "openinference.instrumentation.openai is a shared instrumentor other stacks load alongside their framework (crewai captures prove co-tenancy; smoke-test proves standalone use). No framework seed may claim it (crewai's seed documents why: a crewai rule on this scope would mislabel every OpenAI-SDK span in the fleet).", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "source": "wire", + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.openai" + }, + "owned_by": "library", + "priority": 39982 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.openai" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "C", + "source": "source_code", + "note": "Same OpenInference using_session() context mechanism as the framework instrumentors." + } + ], + "decoy_keys": [], + "decoy_values": [], + "caveats": [ + "Carries the token/model/prompt payload for crewai (and any host framework that drives the OpenAI SDK) — per-span vendor attribution means these spans are openinference-openai even inside another vendor's trace." + ] + }, + { + "vendor": "pydantic_ai", + "seed": "frameworks/pydantic_ai/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "pydantic-ai" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded in models/instrumented.py:133-134 (scope_name = 'pydantic-ai'; get_tracer(scope_name, __version__)) and used for every span the library emits — agent run, model request, tool execution, embeddings, and the concurrency-limiter span. Library-owned and stable; the scope version IS the pydantic-ai package version, so it doubles as a version signal. Catches 19/19 spans in pydantic_ai_user and 20/20 spans in pydantic_ai_agents — after the 2026-08-11 regeneration both captures are 100% native and `pydantic-ai` is the ONLY scope present in either. It is the only signal that classifies `chat` and `execute_tool` spans — see false_negatives.", + "priority": 39981 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "pydantic_ai." + }, + "source": "wire", + "priority": 29953 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "gen_ai.aggregated_usage." + }, + "source": "wire", + "priority": 29952 + } + ], + "session_candidates": [ + { + "key": "gen_ai.conversation.id", + "authority_predicate": { + "op": "present", + "key": "gen_ai.operation.name" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "A" + }, + { + "key": "gen_ai.agent.call.id", + "authority_predicate": { + "op": "present", + "key": "gen_ai.operation.name" + }, + "validation": [ + "non_empty" + ], + "granularity": "run", + "verdict": "A" + } + ], + "decoy_keys": [ + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "9/19 + 9/20 `chat` spans, 18 distinct values for 18 spans (gen-- from OpenRouter). Per model call, never repeats across turns." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id (call_… / toolu_…), 2/19 + 6/20 spans, unique per call. Pydantic AI substitutes pyd_ai_ when the provider supplies none (_utils.py:543-551). Correlates a tool call with its result inside gen_ai.input.messages, not sessions." + }, + { + "key": "service.instance.id", + "source": "wire", + "why": "RESOURCE attribute, generated per process by the OTel Python SDK (not by Pydantic AI): one uuid4 on all 19 spans of pydantic_ai_user and another on all 20 of pydantic_ai_agents. Perfectly session-shaped in a fixture where one process is one scenario — and in pydantic_ai_agents it is the ONLY key that spans both traces, which is exactly the trap: it would join the pipeline correctly here and join millions of unrelated sessions in a long-lived server. The archetypal process-identity trap." + }, + { + "key": "gen_ai.agent.name", + "source": "wire", + "why": "Also baggage-propagated to every child span (AGENT_NAME_BAGGAGE_KEY), so it looks like a per-run join key. It is a static agent NAME: one value ('user_assistant') for all 19 spans of pydantic_ai_user, 5 constants in pydantic_ai_agents. Constant forever per deployed agent." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [ + { + "key": "capture.id" + } + ], + "caveats": [ + { + "id": "conversation_id_is_per_run_by_default", + "text": "gen_ai.conversation.id is on ~every span and is a FRESH UUID7 PER RUN whenever the customer passes neither conversation_id= nor message_history= (_agent_graph.py:231-255, branch 4; GraphAgentState.conversation_id defaults to str(uuid7())). Presence is worthless as evidence of a session. Two red flags maple can compute cheaply, neither expressible in the seed algebra: (1) the value is a UUID7 — customer-chosen ids usually are not; (2) it partitions a trace exactly like gen_ai.agent.call.id (the run id), which is the definitive tell. pydantic_ai_agents shows 4 conversation ids inside ONE trace (5 across its 2 traces, none shared); pydantic_ai_user shows 1 across 8 traces. Treat a conversation id that never appears in more than one trace AND coincides with a run id as a run, not a session." + }, + { + "id": "agent_delegation_fragments_the_conversation", + "text": "Sub-agents invoked as tools (the standard Pydantic AI delegation pattern) run with their own message_history, so each delegated worker mints its OWN conversation id — 3 of the 4 values inside pydantic_ai_agents' orchestrator trace. The orchestrator's own id never propagates down. §3's root-most-anchoring picks the orchestrator's value, which is correct behaviour but still only a per-run UUID7 here. Multi-agent Pydantic AI apps are session-fragmented by construction unless the customer forwards conversation_id into every delegated run." + }, + { + "id": "baggage_is_in_process_only", + "text": "The propagation mechanism is OTel BAGGAGE attached inside the agent-run span (capabilities/instrumentation.py:178-181) and read back into each child span's creation attributes. It is in-process context, not W3C baggage headers: nothing is injected into outbound HTTP, so a span produced by a downstream service — or by any library that does not go through Pydantic AI's own instrumentation, including httpx/openai auto-instrumentation nested under `chat` — carries no session key. Conversely, any span opened by CUSTOMER code inside an agent run does NOT get the keys either: only Pydantic AI's own three span types splice the baggage in." + }, + { + "id": "native_error_span_is_real", + "text": "Unlike most of the corpus, the ERROR status and the `exception` span event on `execute_tool fetch_transport_data` (pydantic_ai_agents) are NATIVE: the scenario raises the framework's ToolFailed and capabilities/instrumentation.py records the exception and sets StatusCode.ERROR. error.type is NOT emitted — status + event are the only error signal, so a present(error.type) rule finds nothing here." + }, + { + "id": "hitl_is_invisible_to_span_rules", + "text": "The native declarative approval gate (@agent.tool_plain(requires_approval=True) + DeferredToolRequests) emits NO span for the proposal and NO span for the denial — Pydantic AI short-circuits before the instrumented tool wrapper. In pydantic_ai_user the 2 approval proposals and the 1 denial are visible only inside the `final_result` / gen_ai.input.messages JSON; only the APPROVED call produces an `execute_tool delete_file` span (1/19). The alternative path — ApprovalRequired raised from inside a tool body — does produce a span carrying pydantic_ai.tool.deferral.name, and at instrumentation version < 5 that span is additionally marked ERROR with an exception event." + }, + { + "id": "logfire_keys_are_not_a_pydantic_ai_signal", + "text": "logfire.json_schema on 100% of native spans and logfire.msg on the agent/tool spans are emitted with the Logfire SDK ABSENT — they are a dialect, not a vendor tag. Tempting as a scope-loss fallback (they cover every span, unlike pydantic_ai.*), but they would claim every span of any Logfire-instrumented application. See algebra_violations." + }, + { + "id": "gen_ai_system_is_the_provider", + "text": "gen_ai.system and gen_ai.provider.name both hold 'openrouter' on `chat` spans — the provider, not the framework and not the model vendor. The actual models are openai/gpt-4o-mini and anthropic/claude-haiku-4.5, readable only from gen_ai.request.model / gen_ai.response.model. There is no gen_ai.system value that identifies Pydantic AI." + }, + { + "id": "a_pipeline_is_several_disconnected_traces", + "text": "There is NO framework-level notion of a trace spanning two top-level runs. pydantic_ai_agents' orchestrator run and summary run are two roots, two traces, two conversation ids, two run ids, and nothing joins them span-locally (only the process-level service.instance.id, a decoy). Whatever the customer's mental model, an N-step Pydantic AI pipeline is N traces unless the customer opens their own parent span or threads conversation_id/message_history through every step. This is the corrected fixture: until 2026-08-11 the scenario opened a harness root span (`research_briefing_pipeline`, scope trace-capture.pydantic-ai, briefing.* attrs) that fused the two into one 21-span trace and hid this fact. That span and its attributes are gone; both captures are now 100% native, with zero harness spans and zero harness-mutated spans." + }, + { + "id": "all_values_are_strings_on_the_wire", + "text": "Numbers and booleans arrive as OTLP stringValue via the OTel Python SDK's normal typing only where the framework passes strings; note in particular gen_ai.response.finish_reasons is an ARRAY attribute (canonicalized to '[\"stop\"]'), while gen_ai.input.messages / gen_ai.output.messages / pydantic_ai.all_messages / gen_ai.tool.definitions / model_request_parameters / final_result are JSON-encoded STRINGS, not structured OTLP values. final_result is raw text when the output is a str and JSON otherwise — the same key holds two formats." + } + ], + "variants": [ + { + "name": "instrumentation-version-2", + "trigger": "InstrumentationSettings(version=2) — explicit opt-in; default is 5 (_instrumentation.py:32 DEFAULT_INSTRUMENTATION_VERSION). Emits a PydanticAIDeprecationWarning but is still accepted in 2.24.0 (models/instrumented.py:140-148 accepts 2, 3, 4, 5).", + "effects": "InstrumentationNames.for_version (_instrumentation.py:550-576) rewrites names AND keys: span names become the constants `agent run` / `running tool` / `running output function` instead of `invoke_agent {agent}` / `execute_tool {tool}`, so every span-name heuristic breaks; the agent name moves to `agent_name` only (gen_ai.agent.name is not set by the run span); tool payload keys become `tool_arguments` / `tool_response` instead of gen_ai.tool.call.arguments / gen_ai.tool.call.result. Unaffected: scope.name, the pydantic_ai.* namespace (pydantic_ai.all_messages is version-independent), gen_ai.operation.name, and the three baggage keys — so this seed's matchers and session candidates all survive.", + "exercised_in_captures": false + }, + { + "name": "instrumentation-version-2-3-4-deferral-as-error", + "trigger": "InstrumentationSettings(version=2|3|4) — any version below 5.", + "effects": "capabilities/instrumentation.py:407-409: a CallDeferred / ApprovalRequired raised from a tool body records an `exception` span event and sets status=ERROR on the execute_tool span, on top of the pydantic_ai.tool.deferral.name attribute. At version 5 the span stays UNSET because deferrals are control flow. Changes span_event_names_observed and makes native HITL indistinguishable from a real failure. (Orthogonal to the declarative requires_approval=True path used in pydantic_ai_user, which emits no execute_tool span at all in any version.) Version 4 additionally changed multimodal content shape inside gen_ai.input.messages (messages.py:1019,1065): type=uri/blob vs the pre-4 shape.", + "exercised_in_captures": false + }, + { + "name": "aggregated-usage-off", + "trigger": "InstrumentationSettings(use_aggregated_usage_attribute_names=False); default True (models/instrumented.py:75).", + "effects": "capabilities/instrumentation.py:255-260 stops rewriting gen_ai.usage.* to gen_ai.aggregated_usage.* on the agent-run span, so invoke_agent carries gen_ai.usage.input_tokens/output_tokens — identical key names to the child `chat` span, which makes naive token rollups double-count, and removes this seed's key_prefix(gen_ai.aggregated_usage.) attr matcher (the pydantic_ai.* matcher still covers the same spans).", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "semantic_kernel", + "seed": "frameworks/semantic_kernel/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "semantic_kernel.utils.telemetry.agent_diagnostics.decorators" + }, + "owned_by": "library", + "source": "wire", + "justification": "get_tracer(__name__) at agent_diagnostics/decorators.py:33 — the scope name IS the library module path, so it is library-owned, vendor-namespaced and cannot be produced by app code. Emits every `invoke_agent ` span (8/28 user, 5/47 agents). No non-SK span can land here.", + "priority": 39980 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "semantic_kernel.utils.telemetry.model_diagnostics.decorators" + }, + "owned_by": "library", + "source": "wire", + "justification": "get_tracer(__name__) at model_diagnostics/decorators.py:67. Emits `chat.completions ` / `chat.streaming_completions ` (10/28 user, 11/47 agents) and, untested here, `text.completions` / `text.streaming_completions`. Library module path; library-owned.", + "priority": 39979 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "semantic_kernel.functions.kernel_function" + }, + "owned_by": "library", + "source": "wire", + "justification": "trace.get_tracer(__name__) at functions/kernel_function.py:48. Emits every `execute_tool -` span (2/28 user, 9/47 agents) — the only spans in the corpus carrying error.type / ERROR status / exception events. Same module also owns the meter (kernel_function.py:49).", + "priority": 39978 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "semantic_kernel.connectors.ai.chat_completion_client_base" + }, + "owned_by": "library", + "source": "wire", + "justification": "get_tracer(__name__) at chat_completion_client_base.py:32. Emits the `AutoFunctionInvocationLoop` span (8/28 user, 5/47 agents), whose ONLY attribute is sk.available_functions — no gen_ai.* key at all, so without this scope matcher (or the key_prefix(sk.) attr matcher) those spans are unclassifiable. NOTE this span is NOT behind the diagnostics gate (see variants.diagnostics_off).", + "priority": 39977 + }, + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "agent_runtime InProcessRuntime" + }, + "owned_by": "library", + "source": "wire", + "justification": "The pathological one: a scope name containing a literal SPACE, with no vendor namespace at all. Built at agents/runtime/core/telemetry/tracing.py:38 as f\"agent_runtime {instrumentation_builder_config.name}\", where the name comes from MessageRuntimeTracingConfig(\"InProcessRuntime\") — a HARDCODED string literal at agents/runtime/in_process/in_process_runtime.py:182, NOT the runtime class name. InProcessRuntime is the only CoreRuntime implementation shipped in 1.36.0, so this eq() is exact and complete for the package as shipped. Marked sufficient because nothing else can classify these 17 spans (17/47 agents, 0/28 user): they carry only messaging.operation / messaging.destination / messaging.message.type, and a generic messaging.* rule would sweep in every Kafka/RabbitMQ span in a real app. Collision risk is real but one-directional and accepted — TraceHelper and MessageRuntimeTracingConfig are PUBLIC exports (agents/runtime/core/telemetry/__init__.py), so any third-party runtime named InProcessRuntime that reuses them would be misattributed to SK; conversely a third-party runtime under any other name is missed (see algebra_violations). These spans are SK spans but NOT GenAI spans — see false_positives. They are also the population that fragments the agents capture into 10 traces and the only spans in the corpus carrying span LINKS (13/17) — see caveats.runtime_fragments_traces_and_stitches_with_links.", + "priority": 39976 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "sk." + }, + "source": "wire", + "priority": 29951 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.operation.name", + "value": "chat.streaming_completions" + }, + "source": "wire", + "priority": 29950 + } + ], + "session_candidates": [], + "decoy_keys": [ + { + "key": "gen_ai.agent.id", + "source": "wire", + "why": "THE decoy. Stable across traces and therefore session-shaped on the wire — one value (e97df81c-c307-4e24-99ef-9b490658f3bc) on all 8 invoke_agent spans across all 8 traces of semantic_kernel_user — but it is Agent.id, a default_factory=lambda: str(uuid.uuid4()) field on the Agent MODEL (agents/agent.py:267), i.e. per-agent-OBJECT identity, customer-overridable and constant for the object's whole lifetime. In a long-lived server one agent object serves every user, so joining on it merges all sessions into one; in semantic_kernel_agents it does the opposite, splitting a single logical run across 5 distinct values (one per worker). It also has no relationship to the thread that actually carries conversation state." + }, + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "Provider completion id (gen-1786460114-rK97L7Wm07Nf3VN3cKyX); unique per LLM call, 10 distinct values in 8 user traces." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id (call_… / toolu_…). Correlates a tool call with its assistant message, never sessions. Retries do NOT reuse it: the 7 failing `execute_tool TransportPlugin-fetch_transport_data` spans in semantic_kernel_agents carry 7 distinct ids, so it cannot group retry attempts either." + }, + { + "key": "messaging.destination", + "source": "wire", + "why": "Looks like a durable session/actor address and embeds a 32-hex GUID (weather_worker_fd657f72c9b14485b2f54f50c2ee41bc.(ConcurrentOrchestration)-A), and it is the ONLY attribute shared across the 10 fragmented traces of semantic_kernel_agents — the single most tempting stitch key in the corpus. But the GUID is the orchestration instance minted per ConcurrentOrchestration.invoke() and the string is an AgentId/TopicId rendering — per-run and per-actor, and absent from every non-runtime span." + }, + { + "key": "service.instance.id", + "source": "wire", + "why": "A uuid4 minted by the OTel-Python SDK per PROCESS (eba7a1b6-8670-4d18-91be-a06eb27e0dab). Constant across all 8 traces of semantic_kernel_user, which makes it the most session-shaped value in the capture — and it is pure process identity that would merge every concurrent user of a server into one 'session'. Not SK's; listed because with no real session key present it is the first thing a resolver would reach for." + }, + { + "key": "CHAT_MESSAGE_INDEX", + "source": "wire", + "why": "LOG-record attribute only. An SK-proprietary within-request message ordinal (0,1,2,…) used to restore chat order; not an identifier of anything." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "import_order_silent_zero", + "text": "The diagnostics env vars must be set BEFORE the first `import semantic_kernel`. MODEL_DIAGNOSTICS_SETTINGS = ModelDiagnosticSettings() is a module-level constant in three modules (model_diagnostics/decorators.py:34, agent_diagnostics/decorators.py:28, model_diagnostics/function_tracer.py:26) and is never re-read. Setting them later produces zero GenAI spans with no warning, no log line and no error — and, because AutoFunctionInvocationLoop and the agent_runtime spans are ungated, the trace is NOT empty, it is just silently GenAI-less. Expect customer reports of 'SK sends telemetry but no LLM data'." + }, + { + "id": "gen_ai_system_is_always_openai", + "text": "gen_ai.system is the CONNECTOR class constant, not the provider: OpenAIChatCompletionBase.MODEL_PROVIDER_NAME = \"openai\" (open_ai_chat_completion_base.py:57) is passed to the decorator, so the anthropic/claude-haiku-4.5 spans in semantic_kernel_agents also report gen_ai.system=openai. 12/12 chat spans in the corpus say openai. Provider attribution keyed on gen_ai.system is silently wrong; gen_ai.request.model carries the truth. This is also why eq(gen_ai.system, \"openai\") must NEVER be a semantic_kernel matcher — it would steal every genuine OpenAI-SDK span." + }, + { + "id": "off_semconv_values", + "text": "SK's values are not semconv values. gen_ai.operation.name is `chat.completions` / `chat.streaming_completions` (semconv says `chat`); gen_ai.response.finish_reason is a Python enum repr, `FinishReason.STOP` / `FinishReason.TOOL_CALLS` (semconv says `stop` / `tool_calls`), produced by \",\".join(str(fr)) at model_diagnostics/decorators.py:417-421 — note it is singular `finish_reason` holding a comma-joined LIST, not the semconv plural array. server.address is a full URL, https://openrouter.ai/api/v1/, not a host, and server.port is never set. Every value arrives as OTLP stringValue except the two token counts and max_tokens, which arrive as intValue." + }, + { + "id": "falsy_attributes_are_dropped", + "text": "_get_completion_span filters execution settings with a bare `if attribute:` over extension_data (model_diagnostics/decorators.py:355-358), so any ZERO-valued knob is silently absent: temperature=0 was set in both scenarios and appears on no span, and top_p=0 / seed=0 / frequency_penalty=0 would behave identically. _set_completion_response does the same for usage (`if usage.prompt_tokens:`), so a genuine 0-token count is dropped rather than recorded as 0. Absence of gen_ai.request.temperature does NOT mean the caller left it default." + }, + { + "id": "duplicate_exception_events", + "text": "Failing tool spans carry the `exception` event TWICE — once from KernelFunction._handle_exception's record_exception (kernel_function.py:403) and once from the span context manager's __exit__ as the exception escapes. Wire- confirmed on all 7 `execute_tool TransportPlugin-fetch_transport_data` spans (14 events for 7 exceptions). error.type and the ERROR status are set once each. Any error-event counter must dedupe. Failing tool spans also carry NO gen_ai.tool.call.result at all (2/9 execute_tool spans in semantic_kernel_agents have one), so result presence is not a proxy for tool completion." + }, + { + "id": "error_type_shape_differs_by_path", + "text": "SK has three error paths with two different error.type shapes: kernel_function.py:404 and agent_diagnostics/decorators.py:213 use type(e).__name__ (observed: `RuntimeError`), while model_diagnostics/decorators.py:453 (_set_completion_error) uses str(type(error)), which would emit the literal ``. Only the first shape is exercised in these captures (no chat span errored). A value-keyed error taxonomy must expect both." + }, + { + "id": "tool_errors_do_not_propagate", + "text": "Kernel._inner_auto_function_invoke_handler swallows tool exceptions and feeds the model a string, so the `execute_tool` span is ERROR while every ancestor (AutoFunctionInvocationLoop, invoke_agent, and the whole trace) stays UNSET. Trace health computed from the root, or from status alone, reports these traces as clean: 7 ERROR spans sit under an UNSET root in the 31-span trace. The model retried fetch_transport_data 7× despite instructions not to (5 LLM round trips)." + }, + { + "id": "high_cardinality_span_names", + "text": "Every native span name is templated with unbounded data: `invoke_agent `, `chat.completions `, `execute_tool -`, and worst, `agent_runtime process _<32-hex orchestration GUID>.(Orchestration)-A` — the GUID is minted per orchestration run, so the 17 runtime span names in this capture are unique to this run and will never recur. Span-name grouping, dashboards and any eq(span.name, …) rule are unusable for this framework; that is why this seed contains no span.name predicate at all." + }, + { + "id": "runtime_fragments_traces_and_stitches_with_links", + "text": "THE headline fixture fact, revealed by the 2026-08-11 §7 regeneration (the previous capture hid it behind a harness root span). SK's InProcessRuntime does NOT continue the caller's trace across the actor message bus: TraceHelper.trace_block computes `context = None` — literally, with a `# TODO(evmattso): we may need to remove other code for using custom context.` beside it (agents/runtime/core/telemetry/tracing.py:82) — and passes it to start_as_current_span. Whenever the enqueue and the handling of a message sit in different asyncio tasks (which is always for send/ack, and for the whole bus once no customer span is ambient), the span starts a BRAND NEW TRACE. Instead of parenting, the runtime attaches LINKS built from the envelope's traceparent (get_telemetry_links, propagation.py). Measured: one ConcurrentOrchestration run = 10 traces, 4 of them a single span, with 13/17 runtime spans carrying exactly one link each. Consequences: (a) a customer's own root span makes this LOOK like one clean trace, so the shape maple sees depends on whether the app wraps the call; (b) trace-level anything (duration, error rollup, cost per request) is wrong for orchestrated SK by construction; (c) any stitching must follow span links, which no predicate in the algebra can express." + }, + { + "id": "hitl_is_invisible_on_the_wire", + "text": "The fixture exercises both approval branches and NEITHER is distinguishable in the telemetry. Turn 5 (deny): gpt-4o-mini asks for confirmation in prose, so no tool call is ever emitted and the FUNCTION_INVOCATION filter never runs — the denial exists only in the model's text, which lives in a gen_ai.choice LOG record. Turn 6 (approve): the filter runs and the tool executes, producing an ordinary OK-status `execute_tool FileSystemPlugin-delete_file` span identical in shape to any other tool call. A denied-then-executed gate is therefore indistinguishable from a plain tool call, and a gate that blocks is indistinguishable from a model that simply chose not to call the tool. (The 2026-08-06 capture appeared to show a denial only because the harness emitted an approval.decision span for it — a §2 violation now removed.)" + }, + { + "id": "no_native_hitl", + "text": "SK Python has no tool-approval / interrupt / resume primitive at all — no analogue of LangGraph's interrupt or ADK's LongRunningFunctionTool. AutoFunctionInvocationContext.terminate can stop the loop but offers no pause/resume checkpoint. The idiomatic interception point is a FilterTypes.FUNCTION_INVOCATION filter, which runs INSIDE KernelFunction.invoke — so a filter denial would still produce a full, OK-status execute_tool span whose gen_ai.tool.call.result is the refusal text (unexercised in this capture: the model asks in prose before ever calling the tool, see caveats.hitl_is_invisible_on_the_wire). HITL is invisible to any rule in this algebra, and a denied tool call is indistinguishable from a successful one." + }, + { + "id": "keyword_args_empty_the_input", + "text": "gen_ai.agent.invocation_input is [] whenever the caller passes messages by keyword: the decorators do `messages = args[1] if len(args) > 1 else None` (agent_diagnostics/decorators.py:76,111,147), and `agent.get_response(messages=x)` is the form the SK documentation shows. Both scenarios pass positionally on purpose; a real customer following the docs ships empty inputs while the outputs populate normally. Value-quality caveat, not a matcher." + }, + { + "id": "three_scopes_vanish_by_default", + "text": "Restating the sharpest operational fact: with the shipped defaults three of the five scopes never emit. A customer who wires an OTLP exporter and no env vars sends semantic_kernel traces consisting solely of AutoFunctionInvocationLoop and agent_runtime spans — vendor-classifiable, GenAI-empty, and with gen_ai.operation.name absent the generic unknown tier catches nothing either." + } + ], + "variants": [ + { + "name": "diagnostics_off", + "trigger": "Neither SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS nor SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS_SENSITIVE set to true before the first `import semantic_kernel`. THIS IS THE SHIPPED DEFAULT: ModelDiagnosticSettings declares both fields `= False` (utils/telemetry/model_diagnostics/model_diagnostics_settings.py:30-31) and the settings object is instantiated as a module-level constant in three modules, so the env is read exactly once at import.", + "effects": "Removes THREE of the five scopes entirely — every span from semantic_kernel.utils.telemetry.agent_diagnostics.decorators, semantic_kernel.utils.telemetry.model_diagnostics.decorators and semantic_kernel.functions.kernel_function disappears, i.e. all gen_ai.* spans and all gen_ai.* log records. What SURVIVES is ungated and still classifies as semantic_kernel: `AutoFunctionInvocationLoop` (chat_completion_client_base.py:137,256 — no diagnostics check anywhere on that path) and every `agent_runtime *` messaging span (TraceHelper.trace_block has no gate). A default-configured SK app therefore ships a semantic_kernel trace containing 0% GenAI payload: sk.available_functions and messaging.* only. Also flips fallback_fingerprints to empty (gen_ai.operation.name is gone).", + "exercised_in_captures": false + }, + { + "name": "diagnostics_non_sensitive", + "trigger": "SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS=true with ..._ENABLE_OTEL_DIAGNOSTICS_SENSITIVE unset/false (the documented privacy-preserving setting).", + "effects": "Spans keep their shape and every classification predicate in this file keeps working, but the content attributes vanish: gen_ai.agent.invocation_input / .invocation_output (agent_diagnostics/decorators.py:_set_agent_invocation_input, _set_agent_invocation_output) and gen_ai.tool.call.arguments / .result (kernel_function.py:266,285,338,369) are all behind are_sensitive_events_enabled(). The ENTIRE gen_ai.* log stream also disappears — _set_completion_input and the gen_ai.choice branch of _set_completion_response are inside the same gate — so signals.logs.ai_data would become false. Nothing in classification or session changes.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "smolagents", + "seed": "frameworks/smolagents/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "openinference.instrumentation.smolagents" + }, + "owned_by": "library", + "source": "wire", + "justification": "The scope name is the instrumentor module's own __name__ and the version is its __version__ — openinference/instrumentation/smolagents/__init__.py:51 does trace_api.get_tracer(__name__, __version__, tracer_provider). Library-owned, namespaced, stable, and NOT app-derivable: the customer supplies only a TracerProvider. It is also the ONLY signal that classifies 34/40 (user) and 28/33 (agents) native spans — the LLM / TOOL / CHAIN populations carry no smolagents-specific attribute whatsoever (see false_negatives), so an insufficient scope matcher would leave 85% of this vendor's spans unattributed. Exactly four span-creation sites exist in this scope (MultiStepAgent.run, {Code,ToolCalling}Agent ._step_stream, .generate(_stream), Tool.__call__ — __init__.py:56-130), all agent operations. NOTE the scope version tracks the INSTRUMENTOR (0.1.33), never smolagents (1.26.0): the framework version is not on the wire anywhere.", + "priority": 39975 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "smolagents." + }, + "source": "wire", + "priority": 29949 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "op": "present", + "key": "openinference.span.kind" + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "C" + }, + { + "key": "user.id", + "authority_predicate": { + "op": "present", + "key": "openinference.span.kind" + }, + "validation": [ + "non_empty" + ], + "granularity": "user", + "verdict": "C" + } + ], + "decoy_keys": [ + { + "key": "service.instance.id", + "source": "wire", + "why": "Resource attribute auto-detected by the OTel Python SDK 1.44 as a random uuid4 per process (sdk/resources/__init__.py:507-528), not by smolagents or OpenInference. Session-shaped and stable across all 6 turns of smolagents_user ONLY because the fixture is one process per scenario; in a server it is one value for thousands of sessions. Process identity, never session identity." + }, + { + "key": "metadata", + "source": "wire", + "why": "The third slot of the SAME using_attributes call that carries session.id, so it is present on 42/42 and 33/33 spans and looks like a first-class correlation field. It is an opaque customer JSON blob — here {\"scenario\",\"turn\", \"capture.id\"}, i.e. it CHANGES per turn inside one session. Never join on it." + }, + { + "key": "tag.tags", + "source": "wire", + "why": "Fourth slot of the same context call (42/42, 33/33). A customer-supplied label array — here [\"turn-1\"] … [\"turn-6\"] within a single session. Per-call labels, not identity." + }, + { + "key": "llm.output_messages.0.message.tool_calls.0.tool_call.id", + "source": "wire", + "why": "Provider tool-call id (call_…) on LLM spans; also mirrored into the TOOL span's input. Correlates one tool call to its result, changes many times per turn." + }, + { + "key": "openinference.project.name", + "source": "source_code", + "why": "The only OpenInference RESOURCE attribute that exists (semconv/resource/ __init__.py). Set exclusively by `dangerously_using_project`, a notebook helper that patches ReadableSpan.__init__ (_projects.py:10-27), and by Phoenix-style backends. It is a deployment-wide constant — the langsmith.trace.session_name failure mode. Not observed in these captures; never join on it." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [], + "caveats": [ + { + "id": "no_first_party_telemetry", + "text": "smolagents 1.26.0 contains ZERO telemetry code — it imports opentelemetry nowhere, has no tracer/exporter/callback setting, and honours no OTEL_* env var. 100% of smolagents telemetry maple will ever see is authored by openinference-instrumentation-smolagents, a third-party wrapt monkey-patcher. Practical consequences: (a) the framework version is never on the wire — scope.version is the instrumentor's 0.1.33, and service.version=1.26.0 here is harness-set [H]; (b) instrumentation is opt-in application code, so the population of smolagents users maple can see is only those who installed the OpenInference package and built an SDK; (c) span shapes track the instrumentor's release cadence, not smolagents'." + }, + { + "id": "model_subclass_discovery_is_by_export", + "text": "The instrumentor patches Model subclasses by scanning vars(smolagents) at instrument() time (__init__.py:75-84). A customer's own Model subclass — the normal way to wire a private gateway — is NOT in that dict and is therefore NEVER traced: the trace keeps its AGENT / CHAIN / TOOL spans and silently loses every LLM span, along with all token counts and message content. A smolagents trace with no `.generate` span is this case, not an error. Also version-fragile: patching happens once at instrument() time, so classes imported later are missed." + }, + { + "id": "agent_identity_only_in_span_name", + "text": "No agent.name / agent.id / role attribute exists. The agent is identifiable only by splitting `.run` off the span name, and the fallback when an agent has no name= is the Python CLASS name (_wrappers.py:123). Tool spans are worse: the span name is instance.__class__.__name__ (_wrappers.py:653), so every @tool-decorated function produces a span literally named `SimpleTool` — the fixture declares explicit Tool subclasses purely so the names differ. tool.name carries the real tool name in both styles; span names must never be used as rules. See algebra_violations." + }, + { + "id": "smolagents_task_is_the_previous_turn", + "text": "smolagents.task on `.run` is OFF BY ONE when the agent is reused. The wrapper reads agent.task BEFORE calling the wrapped run() (_wrappers.py:74,124-126), and run() is what assigns it — so on a fresh agent the key is ABSENT and on every subsequent turn it holds the PREVIOUS turn's task. Wire-proven in smolagents_user: turn 1 has no smolagents.task, turn 2 carries turn 1's text, … turn 6 carries turn 5's (5/6 spans). Never use smolagents.task as the turn's input — input.value on the same span is correct. In smolagents_agents it is absent from all 5 AGENT spans (every agent is run exactly once)." + }, + { + "id": "token_counts_double_and_undercount", + "text": "llm.token_count.{prompt,completion,total} appears on BOTH the LLM spans and the `.run` AGENT span, where the AGENT value is agent.monitor's run total (_wrappers.py:214-219) — i.e. the sum of its own LLM children. Any per-span rollup over a trace double counts. In the other direction, scenario B's orchestrator.run total covers only the orchestrator's own LLM calls, NOT the managed agents' (each worker has its own monitor and its own AGENT span), so the root span is not a trace total either. Dedupe on openinference.span.kind = LLM." + }, + { + "id": "session_context_is_instrumentor_side_not_otel_context", + "text": "OpenInference's session propagation is NOT OTel baggage and NOT span-attribute inheritance: using_attributes attaches a plain contextvar that only OITracer reads at span creation. A span created with a stock opentelemetry.trace.Tracer inside the very same `with using_attributes(...)` block gets NOTHING. Wire-proven [H] in the pre-prune backup of smolagents_user: run 1's two `human_approval` spans (plain get_tracer, inside the context) carry no session.id/user.id/metadata/tag.tags at all, while run 2's — after hitl.py added an explicit span.set_attributes(dict(get_attributes_from_context())) — carry all four. Any customer-authored or third-party-auto-instrumented span in a smolagents trace is session-blind unless it opts in the same way." + }, + { + "id": "one_trace_per_run_reset_false_does_not_help", + "text": "agent.run(reset=False) preserves MEMORY, not the trace: every run() opens a new root span and therefore a new trace id (six turns of one persistent agent = six unrelated trace ids in smolagents_user). smolagents exposes no run/thread/conversation identifier to the instrumentor either. Cross-turn correlation exists ONLY through the customer-supplied session.id." + }, + { + "id": "error_shape_and_the_unreachable_recovery_event", + "text": "A failed tool produces status=ERROR + one `exception` event on the TOOL span, and status=ERROR + TWO `exception` events on the enclosing `Step ` span (the wrapper's record_exception at _wrappers.py:325 plus the SDK's own on the way out of start_as_current_span). 0.1.33 also contains a softer path — _record_step_error() emits an `agent.step_recovery` span EVENT and sets status OK for AgentToolCallError/AgentToolExecutionError (_wrappers.py:260-286) — but it is unreachable whenever the error propagates, because _finalize_step_span skips it once the status is already ERROR. Wire-confirmed: zero `agent.step_recovery` events and zero OK-status errored steps in either capture; the transport failure in smolagents_agents is an AgentToolExecutionError and still lands as ERROR + 2 exception events. Error-rate consumers must expect exactly this double-count." + }, + { + "id": "no_native_hitl_or_resume", + "text": "smolagents 1.26.0 has no approval/interrupt-and-resume mechanism. MultiStepAgent.interrupt() only flips a flag checked at the top of the next loop iteration and then raises AgentError — it aborts a run, it cannot suspend a pending tool call. There is no needs_approval / checkpointer concept. The fixture's HITL is therefore SIMULATED [H] by subclassing ToolCallingAgent.execute_tool_call (hitl.py): on denial Tool.__call__ is never reached, so NO DeleteFileTool span exists — a real absence, not a dropped span. maple must not expect any native HITL signal from smolagents." + }, + { + "id": "parallel_fan_out_is_real_and_context_safe", + "text": "Scenario B's three managed-agent calls genuinely overlap (weather/budget/transport workers start within ~5 ms of each other inside one 33-span trace): ToolCallingAgent.process_tool_calls dispatches through a ThreadPoolExecutor with copy_context().run(...), and the instrumentor additionally swaps smolagents.local_python_executor.ThreadPoolExecutor for a context-preserving one (__init__.py:105-123). Parent/child links survive the thread hop. Fragile in a different way, though: it degrades to sequential whenever the model emits a single tool call per message, so fan-out shape is model-dependent, not framework-guaranteed." + }, + { + "id": "payload_grows_quadratically_within_a_session", + "text": "Every `.generate` span re-serialises the ENTIRE conversation into llm.input_messages..* (144 message-content triples across 11 LLM spans in smolagents_user) and re-sends every tool's full JSON Schema in llm.tools.. With reset=False the memory grows monotonically, so late turns of a long session cost far more bytes than early ones — the largest single OTLP record here is 118,815 B for one 3-turn batch. OPENINFERENCE_HIDE_LLM_TOOLS and OPENINFERENCE_HIDE_INPUT_MESSAGES are the levers; trace-count sampling is not." + }, + { + "id": "pruned_capture_inventory", + "text": "Both capture dirs originally held TWO runs of their scenario (the capture server is append-only). They were pruned on 2026-08-06 to the canonical later run (scripts/prune-spec.json: smolagents_user keep 3-5, smolagents_agents keep 5-8) and the seqs renumbered from 0. The pre-prune supersets survive only as records.jsonl.bak-2026-08-06T02-32-20-748Z / …T02-30-45-531Z and are NOT part of the fixture. Two throwaway probe captures — smolagents_probe (2 records / 20 spans / 3 traces) and smolagents_agents_probe (6 / 37 / 1) — are development artefacts and are deliberately excluded from this seed." + } + ], + "variants": [ + { + "name": "openinference-genai-semconv", + "trigger": "OPENINFERENCE_ENABLE_GENAI_SEMCONV=true (or TraceConfig(enable_genai_semconv=True)); default is False — openinference/instrumentation/config.py:109,118,259-263.", + "effects": "Dual-emits OTel GenAI semconv keys ALONGSIDE the OpenInference keys at span END (_spans.py:75-79 — OpenInferenceSpan.end() derives gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.*, gen_ai.usage.*, gen_ai.tool.*, gen_ai.input.messages / gen_ai.output.messages from the already-set openinference attributes, and only for keys not already present). Purely additive: no openinference key is removed, so scope_matchers, attr_matchers and every session answer are unchanged. It DOES change co_occurring_families and fallback_fingerprints — a generic gen_ai.* rule starts firing on the same spans, so smolagents must outrank the gen_ai unknown tier. It does NOT create a session candidate: _genai_conversion.py maps no openinference session.id/user.id onto gen_ai.conversation.id (verified — the string 'session' does not occur in that module). Because the keys are written in end(), they are invisible to any head sampler, unlike the openinference keys (see sampling.key_at_root_start).", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "spring_ai", + "seed": "frameworks/spring_ai/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "org.springframework.boot" + }, + "owned_by": "generic", + "source": "wire", + "justification": "This is Spring Boot's GLOBAL Micrometer-tracing scope, not a Spring AI scope. Its version tracks Boot (4.1.0), not Spring AI (2.0.0). Every Micrometer Observation in the process lands here: these captures already show 20 plain HTTP CLIENT POST spans (12 + 8) in the same traces as the AI spans, and a real app adds Spring MVC server spans, RestClient, JDBC and every @Observed method. Classification MUST come from attr_matchers.", + "priority": 29948 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "spring.ai." + }, + "source": "wire", + "priority": 29947 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.system", + "value": "spring_ai" + }, + "source": "wire", + "priority": 29946 + } + ], + "session_candidates": [ + { + "key": "spring.ai.chat.client.conversation.id", + "authority_predicate": { + "op": "eq", + "key": "spring.ai.kind", + "value": "chat_client" + }, + "validation": [ + "non_empty", + "not_in_decoy_values" + ], + "granularity": "session", + "verdict": "B" + } + ], + "decoy_keys": [ + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "On every `chat` span; unique per LLM call, never repeats across turns." + }, + { + "key": "spring.ai.tool.call.id", + "source": "wire", + "why": "Per tool invocation (call_AzE6...); correlates tool calls, not sessions." + } + ], + "decoy_values": [ + { + "value": "default", + "source": "source_code", + "why": "ChatMemory's documented default conversation id. NOT observed on the wire (the convention omits the key when unset), but an integration passing the literal default into the advisor param would produce a process-wide constant that must never be joined on." + } + ], + "harness_keys_fixture_only": [ + { + "key": "chat.session.id" + } + ], + "caveats": [ + "The instrumentation scope identifies SPRING BOOT, not Spring AI: scope.name is org.springframework.boot, scope.version is the Boot version. A Boot app with no Spring AI on the classpath emits spans in the identical scope. Strongest argument in the corpus for attribute-based over scope-based classification.", + "gen_ai.system is the CLIENT implementation, not the provider: `chat` spans for anthropic/claude-haiku-4.5 (via OpenRouter through the OpenAI-compatible client) report gen_ai.system=openai. gen_ai.request.model / gen_ai.response.model carry the truth.", + "ALL attribute values arrive as OTLP stringValue, including numbers and booleans (gen_ai.usage.input_tokens=\"210\", spring.ai.chat.client.stream=\"false\") and gen_ai.response.finish_reasons as the string [\"STOP\"], not an arrayValue. Micrometer KeyValues are String-typed and the bridge does not re-type them.", + "Emitted OTLP is maximally unbatched: ONE resourceSpans block containing ONE scopeSpans block containing ONE span, repeated — the full resource attribute set is duplicated per span.", + "In 1 of the 2 multi-record traces the first record mentioning the trace contains ONLY the POST HTTP span — neither a classifiable span nor the session key — so a first-batch classifier sees that trace as non-AI for one batch.", + "Spring AI produced no ERROR-status span anywhere: the default ToolExecutionExceptionProcessor swallows tool exceptions and feeds the message back to the model, so execute_tool fetch_transport_data ends OK with the 503 text in spring.ai.tool.call.result. The corpus' only ERROR span is harness [H].", + "28 of 101 native spans are advisor bookkeeping (`call`, `tool _calling `, `message_chat_memory`) with no AI payload beyond spring.ai.advisor.name/order. One has a malformed name — the literal string `tool _calling ` (stray underscore, trailing space; Spring AI name-derivation bug). Span-name rules must not be used.", + "Harness spans [H]: agent.* (8 + 5, AgentSpan.java — hardcodes gen_ai.system=spring_ai, gen_ai.operation.name=invoke_agent, agent.role/parent/ task, workflow.* on the orchestrator, chat.session.id/chat.turn.* on agent.assistant), hitl.approval (3 — Spring AI has NO native HITL mechanism), and transport_api.fetch (1). All emitted via Spring's own ObservationRegistry, so they land in the framework scope and are detectable only by provenance." + ], + "variants": [] + }, + { + "vendor": "strands", + "seed": "frameworks/strands/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": true, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "strands.telemetry.tracer" + }, + "owned_by": "library", + "source": "wire", + "justification": "Library-owned and structurally stable: telemetry/tracer.py:113-116 sets self.service_name = __name__ and calls tracer_provider.get_tracer( self.service_name), so the scope name IS the dotted module path of the tracer module inside the strands package. It cannot be influenced by the app, is namespaced, and catches 33/33 + 25/25 spans — the two captures contain NO span from any other scope at all — including the `invoke_graph` root, which carries only 3 attributes. NOTE: get_tracer() is called with no version argument, so scope.version is EMPTY on the wire — scope.version can never be used to detect the SDK release.", + "priority": 39974 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.system", + "value": "strands-agents" + }, + "source": "wire", + "priority": 29945 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.provider.name", + "value": "strands-agents" + }, + "source": "source_code", + "priority": 29944 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "event_loop." + }, + "source": "wire", + "priority": 29943 + } + ], + "session_candidates": [ + { + "key": "session.id", + "authority_predicate": { + "any_of": [ + { + "op": "eq", + "key": "gen_ai.system", + "value": "strands-agents" + }, + { + "op": "eq", + "key": "gen_ai.provider.name", + "value": "strands-agents" + } + ] + }, + "validation": [ + "non_empty" + ], + "granularity": "session", + "verdict": "C" + } + ], + "decoy_keys": [ + { + "key": "event_loop.parent_cycle_id", + "source": "wire", + "why": "Looks like a parent/session pointer and is WRONG under concurrency: event_loop.py:803 writes it into the invocation_state dict, which multiagent/graph.py:833 SHARES across every node running as its own asyncio.create_task. In strands_agents 3 of the 4 spans carrying it name a cycle belonging to a DIFFERENT agent (weather_worker->transport_worker, budget_worker->weather_worker, summary_agent->weather_worker); the 4th is self-consistent only by luck. Correct in the sequential strands_user case (3/3 same agent, same trace). Never join on it, and never use it to reconstruct causality." + }, + { + "key": "event_loop.cycle_id", + "source": "wire", + "why": "Fresh uuid4 per event-loop cycle: 11 distinct values across 8 traces in strands_user, 8 within the single strands_agents trace. Sub-run granularity — changes several times inside one turn." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id. Deliberately REPEATS across the interrupted and resumed copies of the same execute_tool span, which makes it look like a correlation key spanning traces; it correlates one tool call, not a session." + }, + { + "key": "service.instance.id", + "source": "wire", + "why": "Added to the resource by the OTel Python SDK, not by strands: a fresh uuid4 per PROCESS, constant across all 33 spans and all 8 traces of strands_user. The most convincing decoy in this framework — in a one-script-one-session fixture it is indistinguishable from a session id, and in a long-lived server it silently merges every user into one 'session'." + }, + { + "key": "gen_ai.agent.name", + "source": "wire", + "why": "Stable per agent (user_assistant; orchestrator/weather_worker/...), so it looks like a conversation key in single-agent captures. It is a role name shared by every session that agent ever serves." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [ + { + "key": "scenario" + }, + { + "key": "capture.id" + } + ], + "caveats": [ + { + "id": "agent_span_tokens_are_cumulative", + "text": "gen_ai.usage.* on invoke_agent spans is response.metrics.accumulated_usage — the Agent instance's LIFETIME total, not the turn's (tracer.py:788-812). In strands_user the 8 invoke_agent spans read 248, 595, 1031, 2019, 2576, 3190, 3844, 4541 total_tokens, and 4541 is exactly the sum of all 9 `chat` spans. Summing invoke_agent tokens across a session over-counts by ~4x (18044 vs 4541); summing `chat` spans is correct. The gen_ai_use_latest_invocation_tokens opt-in silently inverts which of the two is right, with no attribute to tell them apart." + }, + { + "id": "content_lives_in_span_events", + "text": "In the default flavour 100% of prompts and completions are span EVENTS (gen_ai.system.message / user / assistant / tool.message / gen_ai.choice), with the payload in the event attributes `content` / `message` as JSON-serialised Strands content blocks. Any pipeline that keeps spans but drops span events keeps every token count and loses every message. The history is replayed IN FULL on each `chat` span, so events grow quadratically with turn count: the 9 chat spans of the 6-turn strands_user conversation carry 36 gen_ai.user.message + 36 gen_ai.assistant.message events between them; 233 span events ride 33 spans." + }, + { + "id": "parent_cycle_id_is_corrupt_under_concurrency", + "text": "event_loop.parent_cycle_id is wrong in 3/4 concurrent cases (strands_agents), naming another agent's cycle, because invocation_state is one dict shared by all parallel graph nodes. Recorded as a decoy_key; repeated here because it is the one strands attribute that actively misleads about trace structure rather than merely being useless." + }, + { + "id": "interrupt_and_resume_are_different_traces", + "text": "Strands' native interrupt (BeforeToolCallEvent.interrupt) stops the event loop and returns; resuming requires a NEW agent(...) call, which opens a NEW root trace. 6 turns -> 8 traces. The interrupted `execute_tool` span is exported with status OK and NO gen_ai.tool.status, and a second span with the same gen_ai.tool.call.id appears in the resume trace with the real outcome. Denial is a native ERROR span (code 2, message = the cancel_tool string) with gen_ai.tool.status absent on the first copy and the tool body never executed." + }, + { + "id": "gen_ai_system_is_the_framework", + "text": "gen_ai.system is the constant 'strands-agents' on every span — never the model vendor. Both captures actually called OpenRouter (openai/gpt-4o-mini, anthropic/claude-haiku-4.5); only gen_ai.request.model carries provider truth, and it holds the OpenRouter id, not a native OpenAI/Anthropic model id. Convenient for classification (it is our attr_matcher), useless for provider attribution." + }, + { + "id": "invoke_graph_is_a_bare_container", + "text": "The multi-agent root has 3 attributes, an UNSET status, no gen_ai.event.end_time and span kind CLIENT — the only non-INTERNAL span in either capture, and kind CLIENT for a span that makes no outbound call. It is also the only span type that reads its trace_attributes from the Graph object rather than from an Agent." + }, + { + "id": "no_native_http_span", + "text": "Strands emits no span for the actual model HTTP request; `chat` wraps the provider SDK call. Latency attribution below `chat` requires separate HTTP/provider auto-instrumentation, whose spans belong to a different vendor and would sit underneath strands spans in the same trace." + }, + { + "id": "attribute_values_are_properly_typed", + "text": "Unlike the JVM frameworks, numeric attributes arrive as OTLP intValue (gen_ai.usage.input_tokens, gen_ai.server.time_to_first_token) and gen_ai.agent.tools arrives as a JSON STRING, not an arrayValue. Nothing in these captures needs string->number coercion." + } + ], + "variants": [ + { + "name": "gen-ai-latest-experimental", + "trigger": "OTEL_SEMCONV_STABILITY_OPT_IN contains 'gen_ai_latest_experimental' (telemetry/tracer.py:120-122). Never a default in the SDK; AWS deployment guides recommend it for new backends.", + "effects": "Renames the vendor-identifying attribute: _get_common_attributes() (tracer.py:1216-1228) emits gen_ai.provider.name=strands-agents INSTEAD OF gen_ai.system, so the eq(gen_ai.system, strands-agents) attr_matcher stops firing and only the scope matcher and the eq(gen_ai.provider.name, ...) matcher classify. Also replaces the per-message span events (gen_ai.user.message / gen_ai.assistant.message / gen_ai.tool.message / gen_ai.system.message / gen_ai.choice) with a single gen_ai.client.inference.operation.details event carrying gen_ai.input.messages / gen_ai.output.messages / gen_ai.system_instructions — span_event_names_observed changes wholesale.", + "exercised_in_captures": false + }, + { + "name": "gen-ai-tool-definitions", + "trigger": "OTEL_SEMCONV_STABILITY_OPT_IN contains 'gen_ai_tool_definitions' (tracer.py:123, used at tracer.py:724-727).", + "effects": "Adds gen_ai.tool.definitions (serialized JSON schema of EVERY registered tool) to every invoke_agent span. Payload-size only; no classification or session field changes.", + "exercised_in_captures": false + }, + { + "name": "gen-ai-span-attributes-only", + "trigger": "OTEL_SEMCONV_STABILITY_OPT_IN contains 'gen_ai_span_attributes_only', OR — IMPLICIT DEFAULT — the substring 'langfuse' appears in OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / LANGFUSE_BASE_URL (tracer.py:125, 193-202). Any customer exporting to Langfuse gets this without asking for it.", + "effects": "Moves aggregated content from span EVENTS to span ATTRIBUTES, shrinking span_event_names_observed and inflating per-span attribute payload. The Langfuse path additionally stamps langfuse.observation.type=span on agent spans (tracer.py:789-790) — a foreign vendor namespace inside strands' own spans.", + "exercised_in_captures": false + }, + { + "name": "gen-ai-use-latest-invocation-tokens", + "trigger": "OTEL_SEMCONV_STABILITY_OPT_IN contains 'gen_ai_use_latest_invocation_tokens' (tracer.py:124, 791-800).", + "effects": "Changes the MEANING of gen_ai.usage.* on invoke_agent spans from response.metrics.accumulated_usage (cumulative over the Agent instance's whole lifetime — see caveats) to the latest invocation only. Same keys, different semantics: a token rollup is wrong under exactly one of the two settings.", + "exercised_in_captures": false + }, + { + "name": "bedrock-agentcore-runtime", + "trigger": "Deployment on AWS Bedrock AgentCore Runtime with the ADOT auto-instrumentation layer. AgentCore propagates the session through the X-Amzn-Bedrock-AgentCore-Runtime-Session-Id header, mirrored into OTel baggage and stamped onto spans as session.id by the runtime layer, not by strands-agents.", + "effects": "The session.id candidate's customer_action becomes 'nothing' and its presence verdict rises from C to A for that deployment: session.id appears on spans even when the app passes no trace_attributes. Resource also gains AWS/ADOT attributes. This is the ONLY configuration in which strands session identity is free; verified from AWS AgentCore observability documentation, not from the wire.", + "exercised_in_captures": false + } + ] + }, + { + "vendor": "vercel_ai_sdk", + "seed": "frameworks/vercel_ai_sdk/registry-seed.yaml", + "goldens_status": "human_reviewed", + "matchers": [ + { + "class": "scope", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "gen_ai" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded at open-telemetry.ts:117 (trace.getTracer('gen_ai')), so library-owned in the provenance sense — but it is a bare, unnamespaced, unversioned word that no vendor can own: no scope.version, no schemaUrl, no scope attributes (verified on all 57 gen_ai-scope spans here and all 205 across the two eve_slack captures). It is also app-overridable via new OpenTelemetry({tracer}) (see variants custom-tracer). A sufficient rule on the literal string 'gen_ai' would claim every span any library or app ever emits from a tracer with that name. Kept because it is the ONLY clean discriminator between the two dialects; classification comes from attr_matchers.", + "priority": 29942 + }, + { + "class": "scope", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "scope.name", + "value": "ai" + }, + "owned_by": "library", + "source": "wire", + "justification": "Hardcoded at legacy-open-telemetry.ts:157 (and again as the fallback in get-tracer.ts:19). Same argument as 'gen_ai' but worse: a two-letter generic noun that any app hand-rolling an AI tracer would plausibly pick. No version, no schemaUrl, no scope attributes on all 18 spans. Sufficient must be false.", + "priority": 29941 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "key_prefix", + "prefix": "ai." + }, + "source": "wire", + "priority": 29940 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "eq", + "key": "gen_ai.operation.name", + "value": "agent_step" + }, + "source": "wire", + "priority": 29939 + }, + { + "class": "attr", + "sufficient": false, + "predicate": { + "op": "present", + "key": "gen_ai.execute_tool.duration" + }, + "source": "wire", + "priority": 29938 + } + ], + "session_candidates": [], + "decoy_keys": [ + { + "key": "gen_ai.response.id", + "source": "wire", + "why": "9/28 + 9/18 + 9/29 spans, one distinct value per LLM call (gen--); never repeats across turns." + }, + { + "key": "gen_ai.tool.call.id", + "source": "wire", + "why": "Provider tool-call id (call_... / toolu_bdrk_...). Correlates a tool call to its request, not turns to a session." + }, + { + "key": "gen_ai.agent.name", + "source": "wire", + "why": "Set from telemetry.functionId (open-telemetry.ts:274) — agent identity, and a PROCESS-LIFETIME CONSTANT per agent ('user_assistant' on all 8 invoke_agent spans in vercel_ai_sdk_user). Session-shaped only because a fixture runs one agent." + }, + { + "key": "ai.telemetry.functionId", + "source": "wire", + "why": "Legacy-dialect twin of gen_ai.agent.name — same constant, 18/18 spans, including ai.toolCall. Same trap." + }, + { + "key": "resource.name", + "source": "wire", + "why": "assemble-operation-name.ts:15 sets it to telemetry.functionId verbatim, so it is the same constant again under a name that reads like an entity id. Note it is a SPAN attribute literally named resource.name, not an OTel Resource attribute." + }, + { + "key": "operation.name", + "source": "wire", + "why": "Legacy dialect: `${operationId} ${functionId}` e.g. 'ai.generateText user_assistant'. Constant per (operation, agent) pair for the whole process life." + } + ], + "decoy_values": [], + "harness_keys_fixture_only": [ + { + "key": "ai.settings.context.sessionId" + }, + { + "key": "ai.settings.context.scenario" + } + ], + "caveats": [ + "SILENT ZERO: AI SDK v7 removed the per-call experimental_telemetry: {isEnabled: true} switch. Telemetry now requires installing a separate package (@ai-sdk/otel) and calling registerTelemetry(...) once at startup. Without it the SDK emits no spans, no warning and no error. Every v5/v6 integration guide still in circulation produces a silently empty pipeline on v7. This bounds how much vercel_ai_sdk data maple should expect to see at all.", + "TWO DIALECTS, ONE VENDOR: the same library ships two mutually exclusive span schemas with different span names, different tree shapes and different attribute namespaces. scope.name (gen_ai vs ai) is the ONLY clean discriminator: legacy ai.generateText.doGenerate spans also carry gen_ai.system / gen_ai.request.* / gen_ai.response.* / gen_ai.usage.*, so attribute-presence detection fails, and both integrations can be registered at once into one trace.", + "gen_ai.provider.name is the CLIENT ROUTE, not the model vendor: mapProviderName (gen-ai-format-messages.ts:76-109) maps the AI SDK provider string through a well-known-prefix table and falls through to the raw string, so every span here says 'openrouter' even for anthropic/claude-haiku-4.5. ai.response.providerMetadata (opt-in) reveals the real upstream. The legacy dialect writes gen_ai.system=openrouter for the same reason.", + "Span names embed the MODEL ID, never the agent: `invoke_agent openai/gpt-4o-mini`. Two different agents on one model are indistinguishable by name. Agent identity exists only as gen_ai.agent.name, only on the operation span, and only if the customer set telemetry.functionId — ToolLoopAgent's `id` is never exported. Attribution of a `chat` or `execute_tool` span to an agent requires walking the parent chain.", + "DOCS BUG, verified: ai-sdk.dev/docs/ai-sdk-core/telemetry documents runtime context landing as ai.settings.runtimeContext.*. The source has one implementation for both dialects — getRuntimeContextAttributes at supplemental-attributes.ts:114-121 — and it emits ai.settings.context.${key}. All 34 wire occurrences across the two dialects (and all 106 across the two eve_slack captures) use ai.settings.context.*. Anyone building a session rule from the docs joins on a key that does not exist.", + "enrichSpan (open-telemetry.ts:141-156, GenAI dialect only) is a second, undocumented- in-NOTES injection point: it returns arbitrary attributes applied to EVERY AI SDK span type at creation, including `chat` and `execute_tool`, which runtimeContext cannot reach. It is the only way a customer can get a session key onto the token-bearing spans. Its keys are entirely app-chosen, so it is invisible to any closed-key rule.", + "LEGACY DIALECT DROPS THE POST-APPROVAL TOOL SPAN: after a HITL approval resume the tool executes before any step begins; onToolExecutionStart returns early when stepContext is undefined (legacy-open-telemetry.ts:603-604), so vercel_ai_sdk_user_legacy has an ai.toolCall span for get_weather but NONE for the approved delete_file that actually ran. The GenAI integration falls back to stepContext ?? rootContext, so its `execute_tool delete_file` span exists but is parented to `invoke_agent` — which post-§7 is the TRACE ROOT — not to a `step`. A parser assuming tool is a descendant of step gets it wrong. Both behaviours are wire-confirmed here (user: execute_tool get_weather under `step 1`, execute_tool delete_file directly under the root; legacy: ai.toolCall for get_weather only).", + "NATIVE HITL, INVISIBLE TO TELEMETRY: toolApproval: {delete_file: 'user-approval'} is a real v7 gate, but neither dialect emits an approval span, event or attribute. A denied call produces NO execute_tool span at all; the decision survives only inside message payloads ({\"denied\":true,...} in gen_ai.input.messages, output:{\"type\":\"execution-denied\"} in ai.prompt.messages), and approval parts are rendered as bare {\"type\":\"tool-approval-request\"} with no approvalId, tool name or approved flag. HITL is unreachable by any predicate in this algebra.", + "SPAN KIND IS PARTLY USEFUL HERE, unusually: the GenAI dialect emits CLIENT for `chat` (9/28 and 9/29 spans) and INTERNAL for everything else. The legacy dialect emits INTERNAL for all 18. Kind alone still cannot classify, but a kind-based LLM-call rollup works in the GenAI dialect and silently returns nothing in the legacy one. Note the trace ROOT is INTERNAL in both dialects — there is no SERVER span anywhere.", + "ERROR MODELLING IS NATIVE AND CORRECT: the failing tool in vercel_ai_sdk_agents produces execute_tool fetch_transport_data with status ERROR ('transport data service unavailable (503)') and one `exception` span event (exception.type/message/stacktrace), written by @ai-sdk/otel's recordErrorOnSpan. The run does not abort — the error becomes a tool-error part fed back to the model. Unlike google_adk, this exception event is framework-emitted, not harness-injected.", + "NO HARNESS SPANS: the §7 fidelity pass (2026-08-11) deleted the fixture's own tracer (`trace-capture.vercel_ai_sdk`), its `chat_turn {n}` / `research_briefing_run` root spans, the `session.id` span attribute, the `tool_approval.decision` span events and the telemetry.dialect / capture.id resource attributes. Native span counts were unchanged by the removal (28 / 18 / 29 before and after), so the earlier goldens' vendor counts carry over; what changed is the TRACE structure, and it changed a lot.", + "ONE TRACE PER generate() CALL — the AI SDK never joins calls into a trace. With the harness root gone there is no trace-level grouping at all: vercel_ai_sdk_user is 8 traces for 6 user turns, not 6, because a HITL approval resume is a SECOND generate() call and therefore a second, separate trace with its own root. Scenario B is 2 traces, not 1: the orchestrator run (26 spans, workers nested under their delegate `execute_tool` spans) and the summary agent run (3 spans) share nothing. Any app-level notion of turn, conversation or workflow must come from a customer span or from the session key — the trace id will not supply it.", + "UNVERIFIED, no capture and no source in this repo: apps deployed on Vercel with OTel Trace Drains are reported to receive vercel.* attributes alongside the ai.*/gen_ai.* ones. If real, vercel.* is a HOSTING signal, not an AI SDK signal, and must not be added as a vendor matcher — a self-hosted AI SDK app emits none of it.", + "Every attribute value arrives as an OTLP typed value here (intValue/doubleValue/ boolValue as well as stringValue), unlike the Micrometer-bridged frameworks; gen_ai.response.finish_reasons is a real arrayValue. Canonical stringification matters for eq() comparisons on this vendor." + ], + "variants": [ + { + "name": "genai-dialect", + "trigger": "registerTelemetry(new OpenTelemetry(...)) — the integration the package README calls recommended. There is no implicit default: registerTelemetry() must be called with one integration or the other (or both).", + "effects": "scope.name = gen_ai. Span tree `invoke_agent {modelId}` (INTERNAL) -> `step {n}` (INTERNAL) -> `chat {modelId}` (CLIENT) with `execute_tool {toolName}` (INTERNAL) as a sibling of chat under the step. Attributes are OTel GenAI semconv (gen_ai.*) plus the AI-SDK-invented gen_ai.operation.name=agent_step and gen_ai.execute_tool.duration. gen_ai.system is NEVER emitted (gen_ai.provider.name instead). This is the dialect both vercel_ai_sdk_user and vercel_ai_sdk_agents use.", + "exercised_in_captures": true + }, + { + "name": "legacy-dialect", + "trigger": "registerTelemetry(new LegacyOpenTelemetry()) — the pre-v7 span format, kept for observability vendors that parse the old shape.", + "effects": "scope.name = ai (NOT gen_ai). Span tree `ai.generateText` -> `ai.generateText.doGenerate` (which is the step span) with `ai.toolCall` under it; all INTERNAL, no CLIENT span. Every span carries ai.operationId / operation.name / resource.name / ai.telemetry.functionId; `ai.generateText.doGenerate` ALSO carries gen_ai.system=openrouter + gen_ai.request.*/response.*/usage.* — so attribute- presence detection cannot tell the dialects apart, only scope.name can. gen_ai.operation.name is never emitted in this dialect, so the generic gen_ai fallback tier sees only the doGenerate spans (9/18 native spans here). Also drops the post-approval tool span (see caveats).", + "exercised_in_captures": true + }, + { + "name": "genai-supplemental-off", + "trigger": "new OpenTelemetry() with no options — THE DEFAULT for the recommended integration (supplemental-attributes.ts:87-95 sets usage/providerMetadata/ runtimeContext/headers/toolChoice/schema/embedding/reranking all to false).", + "effects": "No ai.* attribute exists on ANY span, so key_prefix(ai.) matches nothing in the GenAI dialect: `chat` and `invoke_agent` spans become false negatives and only `step` (agent_step) and `execute_tool` (gen_ai.execute_tool.duration) stay classifiable. ai.settings.context.* cannot exist at all, so the session candidate is structurally unavailable and every trace is unsessioned. The fixture enables usage/providerMetadata/runtimeContext/toolChoice, which is why the goldens below show full coverage — a default-configured customer does not.", + "exercised_in_captures": false + }, + { + "name": "custom-tracer", + "trigger": "new OpenTelemetry({tracer}) / new LegacyOpenTelemetry({tracer}) — open-telemetry.ts:117 and legacy-open-telemetry.ts:157 both fall back to trace.getTracer('gen_ai'/'ai') only when no tracer is passed. The package README's own example passes tracerProvider.getTracer('gen_ai'), but the name is arbitrary.", + "effects": "scope.name becomes app-chosen, so both scope matchers miss and the only dialect discriminator disappears. Classification degrades to the attr_matchers.", + "exercised_in_captures": false + }, + { + "name": "both-dialects-registered", + "trigger": "registerTelemetry(new OpenTelemetry(), new LegacyOpenTelemetry()) — registerTelemetry is variadic and PUSHES onto globalThis.AI_SDK_TELEMETRY_INTEGRATIONS (ai/dist/index.js:4237-4242); it never replaces, so repeated calls also stack.", + "effects": "Every AI SDK operation produces BOTH span trees in the SAME trace: scope=ai and scope=gen_ai spans as siblings, roughly doubling span count and double-counting every LLM call and tool call. Per-span vendor is still vercel_ai_sdk for both.", + "exercised_in_captures": false + } + ] + } + ] +}