Skip to content

feat(examples): OpenLLMetry (Traceloop SDK) recipe and contract test (TH-8337) - #223

Merged
nik13 merged 11 commits into
devfrom
feat/th-8337-openllmetry
Oct 26, 2026
Merged

nik13 merged 11 commits into
devfrom
feat/th-8337-openllmetry

Conversation

@nik13

@nik13 nik13 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A recipe for OpenLLMetry (the Traceloop SDK) in python/examples/openllmetry. It adds no package, because the SDK already exports OTLP/HTTP. Following decision D-8337, the recipe:

  • points the SDK at fi-collector through environment variables;
  • sets the Future AGI project as resource attributes;
  • turns content capture and metrics off.

The backend already maps traceloop.span.kind in futureagi/tracer/utils/adapters/openllmetry.py (SF-1). This PR adds no second kind table, and the README quotes that map.

  • src/app.py:
    • Setup is a single call: Traceloop.init(app_name, resource_attributes={project_name, project_type: "observe"}, telemetry_enabled=False).
    • The demo app is a @workflow that calls a @task and an @agent; the agent calls a @tool and OpenAI Chat Completions.
    • Everything else comes from the TRACELOOP_* variables, which the SDK reads itself.
    • The app exits if TRACELOOP_API_KEY is set.
  • Pin:
    • traceloop-sdk==0.62.4 (Python >=3.10,<4). The wheel METADATA declares License-Expression: Apache-2.0; there are no Trove classifiers and no license file.
    • 0.62.4 imports httpx and requests without declaring them, so requirements.txt lists both.
    • The SDK is a dependency of this example only.

What the tests prove (real SDK, loopback only)

tests/test_openllmetry_recipe.py runs src/app.py as written, in a subprocess. The real traceloop-sdk 0.62.4 instruments a real openai client pointed at a loopback fake, and exports to the shared harness Receiver. A network guard refuses and logs every non-loopback connection; every run logs zero. All keys are placeholders.

  • Endpoint:
    • A host-only TRACELOOP_BASE_URL gives POST /v1/traces, once per export.
    • A trailing / also gives /v1/traces.
    • /v1/traces is not doubled in 0.62.4.
    • <origin>/tracer gives /tracer/v1/traces.
    • fi-collector serves both paths (fi-collector/pkg/server/server.go:233-234 on future-agi main).
  • Auth:
    • Percent-encoded TRACELOOP_HEADERS="x-api-key=…,x-secret-key=…" arrives as the x-api-key and x-secret-key headers with the exact values. These are the headers fi-collector reads (pkg/auth/interceptor.go:50-53).
    • The secret placeholder contains , and = to prove the encoding works.
    • No Authorization header and no OpenAI key reach the export. Control: an unencoded comma truncates the value.
  • Resource: project_name, project_type=observe and service.name are on every exported resource. fi-collector rejects a batch without project_name (pkg/auth/stamp.go:31-45).
  • Kinds:
    • The decorated spans carry traceloop.span.kind workflow/task/agent/tool.
    • The model call carries gen_ai.operation.name=chat plus the gen_ai.* model/usage keys, and no legacy llm.* keys.
    • All spans form one trace with the expected parent links.
    • A model call outside any decorator has no traceloop.* key, so the backend would not see it as OpenLLMetry.
  • Content:
    • TRACELOOP_TRACE_CONTENT=false leaves no content keys or events, and none of the question, answer or tool output anywhere in the export.
    • Control: unset (the SDK default is on) puts content in gen_ai.input.messages / gen_ai.output.messages / traceloop.entity.input|output.
  • Secrets: no FI key or secret (raw or encoded) and no OpenAI key appears in the export or in the app's output.
  • Metrics: the default calls /v1/metrics, which fi-collector does not route. TRACELOOP_METRICS_ENABLED=false stops it.
  • Opt-in (FI_BACKEND_OPENLLMETRY_PY):
    • Emitted kinds and operation names are keys of the backend's _TRACELOOP_KIND_MAP / _OPERATION_KIND_MAP.
    • The README kind table equals _TRACELOOP_KIND_MAP.
    • Both maps are parsed from the backend file with ast, not copied.

Deviations from the architecture (with evidence; approved: Nikhil 2026-10-03 blanket)

  • The suffix is not doubled. The architecture expected a base URL that already ends in /v1/traces to double the path. In 0.62.4, traceloop/sdk/tracing/tracing.py:366-370 appends the suffix only when it is missing, and the test proves it. The README documents the tested behaviour and still says to set the origin with no path.
  • The header proof is done, not pending. The architecture kept the SDK out of the contract test until CI installed it. This PR installs it and asserts the endpoint and headers instead of marking them pending.
  • No hand-built post_otlp() fixture. The fixture keys the architecture listed (llm.request.model, llm.usage.prompt_tokens, gen_ai.prompt.0.content) are not what 0.62.4 emits:
    • 0.62.4 emits gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.input.messages (openai instrumentor shared/__init__.py:153,283, chat_wrappers.py:552).
    • Whether a key lands in the typed map or in overflow needs a running fi-collector.
    • The real-SDK test replaces the fixture, and that question is recorded as an open SF-1 item below.
  • Python matrix: 3.10, 3.11, 3.12 and 3.13. This covers the PRD's 3.10 + 3.13 and the brief's 3.11 + 3.13.

Review round 1 and fixes (pr-reviewer t_11b6a13d APPROVE, pr-verifier t_ff923d29 VERIFIED at d5bb484, no blocking findings)

All findings were P3. They were fixed before the demo, approved: Nikhil 2026-10-03 blanket.

Finding Commit Fix
R1: the content-rendering note said "not tested", but the backend source already settles it 69d3b7b The README now cites openllmetry.py:114, :214-215, :231 at future-agi main 4af5338. The adapter builds input/output only from indexed gen_ai.prompt/completion keys and then deletes every gen_ai.* key. 0.62.4's gen_ai.input/output.messages are therefore dropped rather than shown. This remains a backend item (SF-1); the recipe keeps content off.
R2: the no-egress checks never proved that the guard logs anything 673c65b Positive control tests/_guard_probe.py makes an IPv4 connect, an IPv6 connect_ex and a DNS lookup to reserved targets. The new test asserts all three are refused and logged, in order.
R3: the header and resource tests passed with zero exports when run alone 673c65b Both now assert that an export arrived.
R4: openai was unpinned 69d3b7b Pinned to openai==3.24.0 in requirements.txt and in the README test command.
R5: 3.10 was advertised but never run 69d3b7b The suite now runs on 3.10.17 and 3.12.10 as well. The README lists 3.10-3.13.
R6: no CI runs the opt-in backend-map pair 69d3b7b The README says when to run the pair. This is a process note; no workflow was added.
V1: the README did not say that the SDK warning for a dropped header pair prints the value 69d3b7b The README now says so.

The R2/R3 changes harden existing tests and pass on both heads: the guard already worked, and the shared fixture already exported.

Tests

Head 69d3b7b, coordinator rerun from the repo root on a clean tree (tickets/TH-8337/exact-head-69d3b7b.txt):

env -u PYTHONPATH PYTHONPATH="python/examples/openllmetry:python:python/tests" uv run --no-project --python <py> \
  --with pytest --with pytest-asyncio --with opentelemetry-api --with opentelemetry-sdk \
  --with opentelemetry-exporter-otlp-proto-http --with requests --with jsonschema --with protobuf \
  --with opentelemetry-proto --with 'traceloop-sdk==0.62.4' --with 'openai==3.24.0' --with httpx \
  pytest python/examples/openllmetry/tests -q -p no:cacheprovider --noconftest -o addopts= -rfEs
                    README command            + FI_BACKEND_OPENLLMETRY_PY (future-agi main 4af5338)
Python 3.10         18 passed, 2 skipped      20 passed
Python 3.11         18 passed, 2 skipped      20 passed
Python 3.12         18 passed, 2 skipped      20 passed
Python 3.13         18 passed, 2 skipped      20 passed
  • The 2 skips are the opt-in backend-map tests.
  • Resolved versions: openai 3.24.0, opentelemetry-sdk / exporter 1.45.0, opentelemetry-instrumentation-openai 0.62.4.
  • No files changed outside python/examples/openllmetry. No secret-like lines were added. The worktree is clean after the run.

Limits and open items

  • Not run against a real fi-collector. Auth 200/401, project stamping, storage, typed-vs-overflow and the rendered kind were not executed. AC-02 is checked only as "emitted kinds are keys of the backend map".
  • Content rendering (open for SF-1). With content on, 0.62.4 writes gen_ai.input.messages / gen_ai.output.messages. The backend adapter builds input/output only from indexed gen_ai.prompt.<i>.* / gen_ai.completion.<i>.* (openllmetry.py:114, :214-215), then deletes every gen_ai.* key (:231). Going by that source, the model-call input/output is dropped rather than shown. This was not run against a backend, and this PR does not change the backend.
  • Spans with both key families. The agent, tool and model-call spans carry both gen_ai.* and traceloop.* keys. They are recorded, not resolved; "one stored kind" is SF-1's assertion.
  • Outside the content switch (from reading the source, not tested):
    • exception messages and stack traces on decorated spans;
    • gen_ai.openai.api_base.
  • Image upload. With content on, a base64 image prompt is uploaded to <base>/v2/traces/..., a route fi-collector does not serve.
  • Coverage gaps. Only the non-streaming OpenAI chat path is exercised. gRPC, streaming and other instrumentors are untested. openai is pinned to the tested 3.24.0.
  • Slow runs. Each run starts the app process 10 times, so a run takes 1-5 minutes on a loaded host.

Docs

The draft docs page (tools-interop/docs/TH-8337-docs.mdx) is corrected to these tested results in future-agi/company-brain #63. It covers the suffix rule, the proven header string, the license, the gen_ai.* keys and troubleshooting.

Video demo

Narrated terminal demo, 6:48, recorded at head 69d3b7b, the head verified in round 2 (pr-verifier t_c85598f0). 1080p H.264/AAC, 12 chapters, burned-in captions. sha256 a0dbccc35282977b5c8d09a66c1a5811a445787b17a2c9d1881559220d30dbb7.

The video, captions, transcript, chapter list, preview and media check are attached privately to Linear TH-8337 (Future AGI workspace access needed).

Every run uses the real traceloop-sdk 0.62.4 and openai 3.24.0 against a loopback OpenAI fake and the shared harness receiver, with placeholder keys. There is no OpenAI, Traceloop or Future AGI call and no live fi-collector.

Time Chapter
00:00 The problem: Traceloop's default endpoint and API-key auth, comma-splitting of TRACELOOP_HEADERS, content and metrics on by default, and how Traceloop span kinds map in Future AGI
00:49 Head 69d3b7b: branch and remote at the same SHA, 0 uncommitted files, 6 commits on 3eaadc8, nothing outside python/examples/openllmetry
01:21 Export: the README ## Run block executed as written. Every export is one POST to /v1/traces; x-api-key/x-secret-key arrive (values not shown), no authorization header; project_name, project_type=observe, service.name on the resource; 5 spans; 0 guard attempts
02:12 Tree: workflow > task, agent > tool, openai.chat, each traceloop.span.kind mapped by the backend's own map file (future-agi main 4af5338) to CHAIN/CHAIN/AGENT/TOOL/LLM; README kind table equals the backend map; the contract test's own 3 functions pass on this run
02:53 Endpoint: all four TRACELOOP_BASE_URL forms land on the paths the README states; /v1/traces is never doubled (tracing.py:365-370)
03:30 Headers: percent-encoded headers arrive intact with a secret containing , and =; CONTROL unencoded: the secret is cut at the comma
04:04 Content: with TRACELOOP_TRACE_CONTENT=false, no content keys and no question/answer/tool-output markers anywhere; unset (SDK default) shows which keys carry them
04:37 Metrics: the SDK default calls /v1/metrics, which fi-collector does not serve; TRACELOOP_METRICS_ENABLED=false stops it
05:00 Guard: positive control for every "guard attempts: 0" (IPv4 connect, IPv6 connect_ex, DNS refused and logged)
05:20 Backend gap (SF-1, not this PR): the future-agi adapter reads only indexed gen_ai.prompt/completion.* and strips gen_ai.*, so 0.62.4's message keys would be dropped; read from source only
05:55 Tests: 20 passed with the backend map file, 0 uncommitted files after, plus the logged matrix above
06:20 Review and limits: round 2 VERIFIED; P3 follow-ups (requirements comment on the unpinned OpenAI instrumentor; the guard control on IPv6-disabled hosts)

Only a 14.3 s frozen test wait was cut; on-screen output matches the coordinator-verified driver output in all terminal chapters.

Coordinator checks:

  • sha256 of the uploaded files, with byte readback of each private upload
  • full decode, exit 0; 12 chapter markers
  • silence: five gaps of 2.1-4.8 s, each at the end of a live terminal take
  • frames from every chapter inspected; placeholders only, no keys or header values in frames, captions or transcript

Stacked on #203 (shared harness, 3eaadc8), base dev. Not merged.

Linear: TH-8337. approved: Nikhil 2026-10-03 blanket

Verification sweep (2026-10-12)

  • Future AGI's collector currently stores Traceloop decorator spans as unknown and does not lift their content into input/output; the README now says so. Fixing that needs a collector change, not this PR.
  • Verified end to end: the real library made real calls, traces were exported to a local Future AGI stack built from dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).
  • Commits added in this sweep:
    • ec57562 docs(examples): state what fi-collector stores for OpenLLMetry spans
  • Merge order: merge test(harness): shared OTLP harness for TH-8103 contract tests #203 first (merge commit), and the README catalog PR docs: list the TH-8103 integrations in the README catalogs #250 last. Everything else is order-independent.

Re-verification (2026-10-16)

Head verified: ec57562 (no new commits; Rick's PR was correct as-is). Docs page: future-agi/docs#1009 (draft, base main).

  • Versions: latest stable traceloop-sdk on PyPI = 0.62.4 = the pin; opentelemetry-instrumentation-openai 0.62.4, openai 3.24.0, opentelemetry-sdk 1.45.1, Python 3.11. Run exactly as the README says against local Future AGI (http://localhost:14338, TRACELOOP_HEADERS percent-encoded, content off, metrics off).
  • Live vendor verification (2026-10-17): real OpenAI Chat Completions (gpt-4o-mini), 3 calls, well under $0.01. No stand-in used for the model.
  • Traces (project th8103-rick-openllmetry-traceloop):
    • content off, success: 16dd8bfe-793d-a03c-9563-50d5ad76683a
    • content on (opt-in): 292fddbb-5217-f398-254f-799dc1b9ea36
    • failure (nonexistent model, 404): 1e228288-bd91-b6c5-f2f5-a5d15dfa7520
  • fi_verify PASS (15 spans, 3 traces). Hierarchy support_request.workflow > normalize_question.task, support_agent.agent > lookup_policy.tool, openai.chat. openai.chat = llm, model gpt-4o-mini, tokens 39/11. Decorator spans stored unknown (as the README states). Failure trace: ERROR on openai.chat, support_agent.agent and support_request.workflow. UI screenshot checked: tree, agent graph and traceloop.span.kind attribute render.
  • Privacy scan (all columns of all 15 span rows): content-off unique marker count 0; content-on marker present on 5/5 spans of that trace only; error-trace question marker 0; no /Users/, username, hostname, Traceback or site-packages. New observation: on the failure trace status_message holds the provider's raw error text (SDK behaviour, not covered by the content switch, already listed in the README); the stored exception event carries name and timestamp only, no stack trace.
  • Tests: python/examples/openllmetry/tests + python/tests/harness: 24 passed, 2 skipped (opt-in). With FI_BACKEND_OPENLLMETRY_PY set to the backend adapter on future-agi dev, the kind-map tests pass.
  • Limitations unchanged: decorator spans unknown until the collector reads traceloop.span.kind (SF-1); input/output columns empty; gRPC untested.

🤖 Generated with Claude Code

nik13 and others added 9 commits October 3, 2026 18:58
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends
no Content-Length. The Receiver read Content-Length only, so every Node
export came back 400 with zero spans; two TH-8103 children had to put a
de-chunking relay in front of it. Receiver now accepts both framings.

It also records one entry per accepted export (path, lower-cased
headers, flattened resource attributes) via requests(), so a contract
test can assert X-Api-Key/X-Secret-Key and project_name through the
shared harness instead of a private recorder.

Verified: 6 harness tests pass, and the TanStack example's real Node
exporter delivered 4 chunked exports (4 spans, collector path, both
auth headers, project_name/project_type) with no relay.

Refs: TH-8339, TH-8103
No traceAI package: traceloop-sdk 0.62.4 already exports OTLP/HTTP.
src/app.py calls Traceloop.init with the project_name/project_type
resource attributes fi-collector needs and wraps one OpenAI call in
@workflow/@task/@agent/@tool. Endpoint, auth headers, content and
metrics come from TRACELOOP_* environment variables read by the SDK.

The app exits if TRACELOOP_API_KEY is set: that key is for Traceloop's
host and the SDK would send it as a bearer token (on every export
without TRACELOOP_HEADERS, and from its image uploader with them).

requirements.txt lists httpx and requests because traceloop-sdk 0.62.4
imports both without declaring them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8337
…Metry

_fake_openai.py serves a non-streaming /v1/chat/completions on
127.0.0.1 with fixed usage and records each request. _guarded_run.py
runs a script with every non-loopback connect/getaddrinfo refused and
logged, so the contract test can show the SDK reached nothing but the
fakes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8337
…dk 0.62.4

Runs src/app.py as written in a subprocess under the loopback guard:
real traceloop-sdk 0.62.4 instrumenting a real openai client against
the fake, exporting to the shared harness Receiver. Asserts:

- a host-only TRACELOOP_BASE_URL posts to /v1/traces once; "/" and
  "/v1/traces" also give /v1/traces (0.62.4 does not append the suffix
  twice) and "/tracer" gives /tracer/v1/traces
- percent-encoded TRACELOOP_HEADERS arrive as x-api-key/x-secret-key,
  with no Authorization header; an unencoded comma truncates a value
- project_name, project_type and service.name on every resource
- workflow/task/agent/tool traceloop.span.kind values, one trace tree,
  gen_ai.operation.name=chat on the model call, and which spans carry
  both gen_ai.* and traceloop.* keys (recorded for SF-1, not resolved)
- TRACELOOP_TRACE_CONTENT=false exports no content; unset, the SDK
  default, exports it
- metrics go to /v1/metrics unless disabled; the app refuses a
  TRACELOOP_API_KEY; no key reaches the export or the output
- a model call outside any decorator has no traceloop.* key

Two opt-in tests read the kind maps from the backend's openllmetry.py
(FI_BACKEND_OPENLLMETRY_PY) instead of copying them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8337
Setup, the endpoint rule (host only; tested paths), the tested
TRACELOOP_HEADERS form for the Future AGI keys, metrics and telemetry,
what is traced, the backend's kind map quoted (checked by the opt-in
tests), the content default (on unless TRACELOOP_TRACE_CONTENT=false),
known gaps, and a Tests section with the exact commands for Python 3.11
and 3.13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
approved: Nikhil 2026-10-03 blanket
Refs: TH-8337
Review round 1 (pr-reviewer t_11b6a13d, pr-verifier t_ff923d29) P3s:
- R2: every no-egress check asserted an empty guard log, but nothing
  showed the guard logs a refused attempt. tests/_guard_probe.py tries an
  IPv4 connect (192.0.2.1), an IPv6 connect_ex (2001:db8::1) and a DNS
  lookup (example.invalid) under _guarded_run.py; the new test asserts all
  three are refused and logged in order.
- R3: the header and resource tests looped over the exports and passed
  with zero exports when run alone; both now assert an export arrived.
Hardening tests: both pass at d5bb484 and here (the guard already worked).

approved: Nikhil 2026-10-03 blanket
Refs: TH-8337
- R1: state what the backend source shows: the OpenLLMetry adapter
  (future-agi main 4af5338, openllmetry.py:114, :214-215, :231) builds
  input/output only from indexed gen_ai.prompt/completion keys and then
  deletes every gen_ai.* key, so 0.62.4's gen_ai.input/output.messages are
  dropped, not shown. Still a backend item (SF-1); recipe keeps content off.
- R4: pin openai==3.24.0 in requirements.txt and the README test command.
- R5: the suite now also ran on Python 3.10.17 and 3.12.10 (20 passed each);
  the README lists 3.10-3.13.
- R6: say no CI runs the example and when to run the opt-in backend-map pair.
- V1: the SDK's warning for a dropped TRACELOOP_HEADERS pair prints the
  whole pair, value included.

approved: Nikhil 2026-10-03 blanket
Refs: TH-8337
@nik13
nik13 requested a review from NVJKKartik October 10, 2026 08:22
A live run of the recipe against fi-collector (future-agi dev ea12cc3)
showed the README's kind claim is wrong for the live ingestion path:
fi-collector does not read traceloop.span.kind, so the workflow, task,
agent and tool spans are stored as unknown. The model call is stored as
llm with model and tokens (from gen_ai.operation.name), and the input and
output columns stay empty because 0.62.4 sets no input.value/output.value.

Say so, scope the quoted Python adapter map to the Django path it runs
on, and drop the advice to keep model calls inside a @workflow, which
only mattered to that adapter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13
nik13 merged commit ddf1544 into dev Oct 26, 2026
4 checks passed
@nik13
nik13 deleted the feat/th-8337-openllmetry branch October 26, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant