Skip to content

test(examples): OpenInference active-ingestion conformance fixture (TH-8336) - #227

Merged
nik13 merged 15 commits into
devfrom
feat/th-8336-openinference
Oct 26, 2026
Merged

nik13 merged 15 commits into
devfrom
feat/th-8336-openinference

Conversation

@nik13

@nik13 nik13 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds an OpenInference active-ingestion conformance fixture at python/examples/openinference/ (Linear TH-8336, decision D-8336). There is no package, no Go port, and no edit to the backend's openinference.py or otel_compat_urls.py.

  • Fixture: tests/fixtures/openinference-trace.otlp.json, one OTLP/JSON request with three spans in one trace:

    • retrieve: openinference.span.kind=RETRIEVER, with the query in input.value;
    • ChatCompletion: LLM, llm.model_name, llm.token_count.prompt/completion;
    • lookup_refund_policy: TOOL.

    The resource carries project_name and project_type=observe.

  • Golden: tests/fixtures/openinference-spans.golden.json, the posted spans verbatim.

  • Tests: tests/test_openinference_fixture.py. It posts the fixture with the shared harness post_otlp() and checks it with compare() against the golden. Four opt-in tests (FI_COLLECTOR_SRC) match the fi-collector source statements behind each source-reading fact, with comments stripped. They match source text and do not run the collector.

  • README: the Django route is retired and is not proof (futureagi/tracer/otel_compat_urls.py:15-16 holds the route only as a comment). It separates what is tested from what is read in source.

What fi-collector does with these keys (source reading at future-agi main 4af5338)

Fixture key fi-collector Stored as
openinference.span.kind 4th kind key (exporter/clickhouse25exporter/converter.go:79-84), lower-cased and validated (:107-131) observation_type retriever / llm / tool
llm.model_name first model alias (pkg/adapter/adapter.go:279-284) model
llm.token_count.prompt/completion first token aliases (adapter.go:296-305), total summed (:230-235) prompt_tokens, completion_tokens
input.value overflow (adapter.go:32-40) and lifted into input (converter.go:343) attributes_extra and input

On an authenticated request, a batch whose resource has no project_name fails with HTTP 400 (pkg/auth/stamp.go:31-46, server.go:458-465); a collector with auth disabled skips that check. A project_name that is not yet a project in the key's workspace is created (pkg/auth/auth.go:202-216). One that still does not resolve is not rejected: its spans are dropped and the request is answered 200, with only a collector-log warning (stamp.go:57-85, server.go:466-468).

Unknown kinds (PROMPT, DECISION, UNKNOWN, empty) are stored as unknown. The function that does this has no error path, so a kind value cannot fail an export. How the trace view renders observation_type is shared processor work (SF-1) and is not claimed here.

Deviations from the spec (approved: Nikhil 2026-10-03 blanket)

  • D1, PRD premise false: the PRD says fi-collector does not map openinference.span.kind. It does, via the converter (not DeriveHotKeys), and did already at d794a49. fi-collector's own test pins "Retriever" → "retriever" (converter_test.go:784). Nothing to fix; only the display remains SF-1.
  • D2: input.value is overflow and lifted into the input column, so no new column is needed.
  • D3, license: the legacy License field of openinference-instrumentation 0.1.70 is null, but the wheel declares License-Expression: Apache-2.0. The README claims no license, as the architecture says.
  • D4, AC-01 round trip (post to a live fi-collector and read back): not done; the architecture limits this card to the harness. As a partial substitute outside the repo tests, the builder ran fi-collector 4af5338's own decoder and ConvertWithIdentities offline on the fixture and on unknown-kind variants. That run used no server, auth or ClickHouse.
  • D5: "an unknown kind must not 4xx" is weak against the harness Receiver, which never reads kinds. The collector side rests on source plus that offline run, and the README says so.
  • IDs must be hex: fi-collector decodes OTLP/JSON with pdata UnmarshalJSON. Base64 IDs (what protobuf MessageToDict produces) fail with 400 (server.go:438-442). A test pins hex IDs.

Review

  • Round 1: pr-reviewer (APPROVE) and pr-verifier (VERIFIED) at 7fd83eb, with no P1/P2 findings. The P3s that made the README claim more than the tests prove are fixed in ba6cc08..ea8859b:
    • R1: the project_name 400 is qualified (authenticated path only), and the drop-with-200 case is stated.
    • V1: the opt-in tests now match the fi-collector statements behind each fact (Split, DeriveHotKeys, resolveObservationType, spanToRow, handleHTTPTraces, StampResourceAttrs, ResolveProjectsForKey), with comments stripped. In the builder's mutation run they catch 27 of 27 source mutations; the 7fd83eb tests caught 10. The README now says they match source text and do not run the collector.
    • R5: an ast scan of every .py file in the example and the harness rejects openinference/tracer imports, including ones that never run. The run-time check stays.
  • Round 2: pr-verifier t_353b2a8e VERIFIED at ea8859b, no P1/P2. Retained P3s (narrated in the video, not fixed here): V2 a failed project auto-create drops the batch with HTTP 200, while lookup/scope failures return 400; V3 project_type is an end-user gate, and its absence is not an error; V4 the import scan does not catch importorskip or dynamic/non-literal imports.
  • Follow-ups outside this PR: the stale spec/docs text (kind already mapped, license) is corrected in company-brain Dev #63 (886fe7d, open); a converter_test.go case that decodes this fixture is future-agi work.

Tests

Head ea8859b. Coordinator rerun from the repo root on a clean tree (tickets/TH-8336/exact-head-ea8859b.txt):

env -u PYTHONPATH PYTHONPATH="python:python/tests" uv run --no-project --python <py> \
  --with 'pytest==9.1.1' --with 'opentelemetry-sdk==1.45.0' --with 'opentelemetry-exporter-otlp-proto-http==1.45.0' \
  --with 'opentelemetry-instrumentation==0.66b0' --with 'requests==2.34.2' --with 'jsonschema==4.26.0' \
  pytest python/examples/openinference/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs
              default                FI_COLLECTOR_SRC=<fi-collector @ 4af5338>
Python 3.10   21 passed, 4 skipped   25 passed
Python 3.11   21 passed, 4 skipped   25 passed
Python 3.12   21 passed, 4 skipped   25 passed
Python 3.13   21 passed, 4 skipped   25 passed
  • 4 files, all under python/examples/openinference/. 0 secret-like added lines; 0 files outside the directory.
  • The tests post only to the harness Receiver on 127.0.0.1. There is no fi-collector, Arize account or live instrumentor.

Coverage:

  • golden match, with a compare() control (a kind's case change fails it);
  • tree and parentage;
  • kind strings as posted, and no other kind key;
  • model and token keys and types, and the query in input.value;
  • hex IDs and the resource project_name;
  • unknown kinds post unchanged;
  • post_otlp() refuses non-loopback endpoints;
  • no openinference or tracer import in the example or harness source (ast scan with a control), and none loaded at run time;
  • no key material in the fixture.

Limits

  • The Receiver asserts what is posted, not what is stored. Auth (401), project stamping, storage and the trace view need a collector, which the harness does not start.
  • The PRD's privacy and tenant checks (two projects) are not in the architecture and are not done.
  • No CI job runs this directory. Run the opt-in tests whenever fi-collector's kind, alias or project code changes. They are text checks: a refactor that keeps behaviour fails them, and a behaviour change outside the pinned statements passes them.

Video demo

Recorded build: head ea8859be850ced2f6b2b6fa80828d07c1986ad01 (this PR); real terminal chapters from a clean read-only worktree; 7:00, 11 chapters, narrated with captions.

The video (SHA-256 8bf39ecd796b5f84c3bd86a18a564a4af23e9e51da73b16096fc91843a63cdfe), captions (SRT), transcript, chapter list, preview, recorded driver output and both media-verification notes are private attachments on Linear issue TH-8336 (Future AGI workspace; sign-in required). They are not linked here as raw upload URLs.

Time Chapter
00:00 Problem and scope
00:33 Verified build and live remote
01:04 Three-span fixture and parent tree
01:39 Real loopback post and negative control
02:19 Pinned source: kind mapping
03:02 Pinned source: model, tokens and input
03:37 Pinned source: qualified project outcomes
04:21 Four opt-in tests and real mutation
04:58 AST import guard and disclosed gaps
05:39 README test commands and clean tree
06:15 Review verdict and remaining limits

Independent media checks (Rick): full audio/video decode exit 0; 11 embedded chapters match the list; the only audio gaps (about 1.5 s) are the 10 chapter transitions; every end-of-chapter frame inspected (loopback port and temp paths shown as placeholders, header names only); 8 delivered files scanned with zero secret/path/port/header hits. Disclosed edits (recorder): two explanatory cards, reading holds, cropped/panned source views, and one chapter re-recorded at a smaller font so its completion marker fits. The video shows the harness Receiver and pinned collector source reading; it does not show a running fi-collector, storage, auth, tenants or the trace view.

Stacked on #203 (shared harness, 3eaadc8), base dev. Not merged.

Linear: TH-8336. approved: Nikhil 2026-10-03 blanket

Verification sweep (2026-10-12)

Re-verification (2026-10-16)

The fixture PR only proved what is posted; this adds the live round trip the architecture deferred.

  • Versions: latest openinference-instrumentation 0.1.71 + instrumentation-openai 0.1.65, and the pinned 0.1.70 + 0.1.64; openai 3.33.0, opentelemetry-sdk/exporter 1.45.1, Python 3.12. Local Future AGI stack on dev, OTLP/HTTP uncompressed to /v1/traces.
  • Real run (python/examples/openinference-live/live_e2e.py): agent > retriever, llm (gpt-5.6-terra via CLIProxy, 355/22 and 408/22 tokens), tool, llm; second trace: tool exception and 400 from an unknown model.
    Trace ids: 79550063370e7a0ef461a2852fa4835c (0.1.71), 3666a8ea8c8412af15cc2beaad44df9a (0.1.70), e7f9a95c2d39c0b9252504fc0667f5d7 (0.1.71, hide flags on).
  • fi_verify (PASS, expect llm/tool/retriever/agent):
    - [agent] support-agent in=58B out=59B
      - [retriever] retrieve-docs in=58B
      - [llm] ChatCompletion model=gpt-5.6-terra tok=355/22
      - [tool] lookup_refund_policy
      - [llm] ChatCompletion model=gpt-5.6-terra tok=408/22
    - [agent] failing-agent
      - [tool] broken_tool status=ERROR:RuntimeError: ...
      - [llm] ChatCompletion status=ERROR:BadRequestError: ...
    
    Same tree on 0.1.70. UI trace tree and Agent Graph checked by eye. The kind-display gap noted earlier (SF-1) is not present on this stack: kinds render as agent/retriever/llm/tool.
  • Privacy (marker in prompt, tool args/result, retrieval query+doc, exception text; scan of every column of the stored spans): defaults store it (input, output, attributes_extra, attrs_string, status_message). With the six OPENINFERENCE_HIDE_* variables all content columns hold __REDACTED__ and the marker is absent everywhere except status_message of the two ERROR spans. That is upstream OpenInference/OTel behaviour (exception text is not covered by the hide flags); documented in the docs page, no collector change proposed here. No stack traces, paths or usernames found.
  • Tests: existing 21 pass (+4 opt-in skipped); the 4 opt-in tests pass against future-agi main 3257f9614 fi-collector source (25 passed). New static tests for the live script: 4 passed.
  • Changed: added python/examples/openinference-live/ (script, pinned requirements, README, tests) and a pointer in the fixture README. No fixture or harness change; Rick's PR needed no fix.
  • Docs page: docs(integrations): OpenInference tracing (TH-8336) docs#1007 (draft, base main, no navigation.ts; suggested nav Integrations > Import > OpenInference).
  • Limits: only the OpenAI instrumentor was run; sessions, embeddings and the hosted collector were not.

Update 2026-10-21: merged dev into the branch (07024f2, no conflicts) so dev's Python tests workflow runs. 3c3be78: the opt-in test_collector_project_name_rules failed against future-agi main b3737c4 (released 2026-10-21). handleHTTPTraces now routes simulation spans before stamping project_name and writes its 200 through writeExportResponse. The behaviour is unchanged: still a 400 without project_name, and an unresolved project is dropped with a 200. The test now matches braces to find the authenticated branch and follows the helper for the 200. It passes with FI_COLLECTOR_SRC at 4af5338, main b3737c4 and dev 674f2f3, and fails when the stamp's 400 is changed to a 200 or when the stamp is moved out of the authenticated branch. The README notes the re-check.

nik13 and others added 12 commits October 3, 2026 18:58
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends
no Content-Length. The Receiver read Content-Length only, so every Node
export came back 400 with zero spans; two TH-8103 children had to put a
de-chunking relay in front of it. Receiver now accepts both framings.

It also records one entry per accepted export (path, lower-cased
headers, flattened resource attributes) via requests(), so a contract
test can assert X-Api-Key/X-Secret-Key and project_name through the
shared harness instead of a private recorder.

Verified: 6 harness tests pass, and the TanStack example's real Node
exporter delivered 4 chunked exports (4 spans, collector path, both
auth headers, project_name/project_type) with no relay.

Refs: TH-8339, TH-8103
Three spans in one trace, OTLP/JSON as fi-collector's HTTP handler
decodes it (hex trace and span ids, int64 values as decimal strings):

- retrieve: openinference.span.kind=RETRIEVER, input.value (the root)
- ChatCompletion: openinference.span.kind=LLM, llm.model_name,
  llm.token_count.prompt, llm.token_count.completion (child of retrieve)
- lookup_refund_policy: openinference.span.kind=TOOL (child of retrieve)

The resource carries project_name, which fi-collector requires
(pkg/auth/stamp.go:31-45), and project_type=observe. The keys are the
ones openinference-semantic-conventions 0.1.41 defines and adapter.go /
converter.go name; nothing imports openinference-instrumentation.

openinference-spans.golden.json is the posted body's spans, verbatim,
not a rendered trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
States that the Django OTLP route is retired and is not proof
(future-agi main 4af5338 otel_compat_urls.py:15-16: the only route is
commented out), and what the fixture posts and why it is encoded as it
is: hex ids, because fi-collector's pdata JSON decoder rejects base64
(server.go:438-442).

Records, as source reading of fi-collector 4af5338, that the collector
already reads every key the fixture sets: openinference.span.kind is the
fourth kind key (converter.go:79-84, lower-cased and validated at
:107-131), llm.model_name and llm.token_count.prompt/completion are the
first aliases (adapter.go:279-305), and input.value is overflow
(adapter.go:32-40) lifted into the input column (converter.go:343).
Kind values outside the type list, including OpenInference's PROMPT and
DECISION, are stored as unknown with no error path (converter.go:127-129,
:407). Display is SF-1 and not claimed.

Pins openinference-instrumentation 0.1.70 for the later round trip only:
not installed, not imported, no license claimed. Tested Pythons are 3.11
and 3.13; 3.10 and 3.12 are stated as untested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
Posts fixtures/openinference-trace.otlp.json with post_otlp() to the
shared harness Receiver and checks the decoded spans with compare()
against the golden, plus exact equality (ids, parentage, times). Every
always-on assertion is about the posted body, since the Receiver does
not authenticate, resolve kinds or store anything:

- one export to /v1/traces with no key headers; the resource carries
  project_name and project_type=observe;
- one trace, retrieve the parent of ChatCompletion and
  lookup_refund_policy; ids are hex as fi-collector's JSON decoder needs;
- kind strings posted verbatim (RETRIEVER, LLM, TOOL) and no other key
  the collector reads a kind from; a control shows compare() fails when
  a kind value's case changes;
- the LLM span's model and intValue token keys, the retriever's query in
  input.value, the tool span's kind only;
- kinds PROMPT, DECISION, "" and not-a-kind post without error and arrive
  unchanged (the docstring says the Receiver never reads kinds);
- post_otlp() refuses non-loopback endpoints before urlopen, with a check
  that the patched urlopen is on the real path; no openinference or
  tracer module is imported; the fixture holds no key material.

Two opt-in tests (FI_COLLECTOR_SRC) read fi-collector's Go tables
(spanKindAttrKeys, knownObservationTypes, spanKindSynonyms, the model and
token alias lists, overflowKeyPrefixes) rather than copying them. They
pass against 4af5338 and fail against copies with the kind key removed or
"prompt" added to the type list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
The four evidence runs on Python 3.11 and 3.13 took 2.7-3.0 s each, not
about a second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
Coordinator exact-head run at 5be86d4 passed on 3.10, 3.11, 3.12 and 3.13
(18 passed + 2 skipped default; 20 passed with FI_COLLECTOR_SRC).

approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
…orts

The sys.modules check could not fail with the README command: neither
openinference nor the backend's tracer package is installed. Add an ast
scan of every .py file in the example and the harness for imports of
either (import, from-import, import_module/__import__ of a literal), with
a control, and keep the run-time check for environments that have them.
The README says what each one guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
…nce facts

The opt-in source tests pinned key tables and literals, so edits that
falsify README facts passed them (verifier V1: project_name fail-fast
disabled, unknown kind returning "", observation_type not from the
resolver, total_tokens derivation removed, kind key moved first).

Read function bodies with comments removed and whitespace collapsed and
match the statements that carry each fact: Split routing, DeriveHotKeys
model/token/total, resolveObservationType's first-key, lower-case,
synonym and unknown fallback, spanToRow's stored columns and single nil
error return, the kind key order, and the project_name statements in
handleHTTPTraces, StampResourceAttrs and ResolveProjectsForKey.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
…the drop

The README said fi-collector fails a batch without project_name with a
400, unqualified. That holds only on an authenticated request: stamping
is gated on auth.FromContext (server.go:458, stamp.go:19-21). It also
left out that a project_name that is created on demand but still does
not resolve is dropped with a 200 and a log warning, not rejected
(auth.go:202-216, stamp.go:57-85, server.go:466-468). State both, and
guard the wording with a README test; the opt-in test reads the lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
The README said the opt-in tests "check that the kind, model, token,
input.value and project_name facts above still hold". They match source
text, not behaviour. List the tables and statements they read, say how
they match, and that a behaviour change outside those statements passes
them while a behaviour-preserving refactor fails them. Guard the wording
with a README test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8336
@nik13
nik13 requested a review from NVJKKartik October 10, 2026 08:22
nik13 and others added 3 commits October 16, 2026 19:56
…AGI collector (TH-8336)

Real openinference-instrumentation 0.1.71/0.1.70 + OpenAI instrumentor, agent/retriever/tool/llm
and two failure cases, marker-based privacy check. Static tests only in CI.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ctor (TH-8336)

The opt-in source test failed against future-agi main b3737c4: handleHTTPTraces now routes simulation spans before stamping project_name, and writes its 200 through writeExportResponse. The behaviour the docs describe (400 without project_name, unresolved project dropped with a 200) is unchanged. The test now reads the authenticated branch by matching braces and follows the helper for the 200. It passes on 4af5338, main b3737c4 and dev 674f2f3, and fails when the stamp's 400 becomes a 200 or the stamp moves out of the authenticated branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13
nik13 merged commit 36db0d5 into dev Oct 26, 2026
8 checks passed
@nik13
nik13 deleted the feat/th-8336-openinference branch October 26, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant