Repository navigation
docs(examples): OpenLIT recipe with key inventory and fi-collector conformance (TH-8338) - #228
Merged
Merged
Conversation
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103
openlit.init() with the collector origin, the Future AGI keys as percent-encoded otlp_headers, content, metrics and events off, and a local empty price table so init() does not download one from GitHub. The project comes from OTEL_RESOURCE_ATTRIBUTES; the app refuses to start without it or without an endpoint. Pins openlit 1.45.0 and openai 2.54.0. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same helpers as the OpenLLMetry recipe: a 127.0.0.1 Chat Completions fake, a runner that refuses and logs every non-loopback connection, and a probe that the positive-control test runs through it. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs src/app.py as written against the shared Receiver and a loopback OpenAI fake, under the network guard. Checks the endpoint forms (origin gets /v1/traces appended; a full path is doubled), x-api-key and x-secret-key from percent-encoded otlp_headers, project_name and project_type from OTEL_RESOURCE_ATTRIBUTES, the exact key inventory and values, content off and on, metrics and events paths, the GitHub price download as the only outbound attempt, @openlit.trace ignoring the content switch, a 401 reply, and OpenLIT plus traceai-openai giving two model-call spans for one call. An opt-in test reads fi-collector's alias lists and checks the README display table against them. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quotes the 1.45.0 init() signature, records the tested endpoint, header and resource behaviour, the key inventory and which keys fi-collector promotes to columns, the SF-1 gaps, what the content switch does not cover, and "enable one" for traceai-openai and Firecrawl. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Coordinator exact-head run at e29826a passed on 3.10, 3.11, 3.12 and 3.13 (25 passed + 3 skipped README command; 26 passed + 2 skipped with FI_COLLECTOR_SRC; 28 passed with traceai-openai 0.1.10 on 3.11 and 3.13). approved: Nikhil 2026-10-03 blanket Refs: TH-8338
Review R1 (P2) and V1. With capture_message_content=False, OpenLIT 1.45.0 still exports model-generated tool calls (gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.args), the caller's `user` (gen_ai.request.user), and, for a failed call, two exception events plus the status description carrying the SDK's error body. New fixtures run the recipe's own init_tracing(): tool_call.py (tools and user; the fake answers with a tool call whose arguments carry a planted email) and failed_call.py (a fake 500 with max_retries=0). The tests assert that these markers DO arrive with content off and that no prompt or answer text does. They document exceptions; they are not privacy claims. A README test fails if Privacy stops naming any key those runs exported. The README summary no longer says "content off" without qualification, and Privacy recommends disabled_instrumentors=["openai"] or accepting the export; the recipe filters nothing (D-8338). approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review R5. init_tracing() read FI_API_KEY and FI_SECRET_KEY with
os.environ[...], so a missing key ended in a KeyError traceback, unlike
the endpoint and project checks. It now checks both next to them and
raises SystemExit("Set FI_API_KEY and FI_SECRET_KEY ..."). The start-up
test covers both keys and asserts no traceback and no key value in
stderr.
approved: Nikhil 2026-10-03 blanket
Refs: TH-8338
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review V2. Nothing tied the README's copy-paste surfaces to src/app.py. A test now extracts the openlit.init(...) keyword arguments from the README code block and from app.py with ast and asserts they are equal, including capture_message_content=False. Another asserts that the Run block exports the variables app.py checks and that its OTEL_RESOURCE_ATTRIBUTES gives a project_name under the parser app.py uses. The README block now shows app.py's call verbatim, with comments naming each variable. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review R2. The README implied that traceAI's register() sets a global tracer provider; by default it does not. tests/register_first.py calls register() before the recipe's init_tracing() and runs the model call inside a register() span, against two loopback receivers. Tested: - register(set_global_tracer_provider=True): openlit.init() reuses it; otlp_endpoint, otlp_headers and OTEL_RESOURCE_ATTRIBUTES are ignored for spans, and all spans go to FI_BASE_URL under register()'s project. - default register(): two providers and two exports, OpenLIT's spans under the OTEL_RESOURCE_ATTRIBUTES project and the register() span under register()'s, still one trace. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ter.go Review R4. The opt-in drift test hardcoded input.value/output.value, so a collector change to the keys behind the input and output columns left the README rows stale while the test passed. It now extracts the keys passed to overflowAsString(...) for input and output from converter.go, expects the README rows to name exactly those keys, checks that none was emitted, and checks that the content-on keys are not among them. Against a converter.go copy with input.value changed to gen_ai.input.messages, the old test passes and the new one fails. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review R3, plus the verifier's addition. New README subsection, each item tested unless marked: - OTEL_EXPORTER_OTLP_TRACES_HEADERS replaces OpenLIT's header string, so the FI keys are dropped and exports get 401 (a recorder that answers 401 without the keys, as fi-collector's auth middleware does). - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT wins over otlp_endpoint, is used as is, and the FI keys go to it. - OTEL_TRACES_EXPORTER=none|console adds no OTLP exporter. - OTEL_EXPORTER_OTLP_HEADERS is overwritten (read from source). - In-process inheritance: an OTLP exporter built later in the same process without headers (tests/second_exporter.py) sends the FI keys to whatever endpoint it was given. A README test fails if the subsection stops naming these variables. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Tests section now names the fake's tool-call and error replies, lists every fixture script (tool_call, failed_call, register_first, second_exporter), lists the new checks, and says a full run starts the app about 30 times. approved: Nikhil 2026-10-03 blanket Refs: TH-8338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…TH-8338) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds an OpenLIT recipe at
python/examples/openlit/(Linear TH-8338, decision D-8338). There is no traceAI package and no OpenLIT adapter. OpenLIT 1.45.0 already exports OTLP/HTTP; the recipe pointsopenlit.init()at fi-collector and measures what arrives.What the recipe sets (
src/app.py):otlp_endpoint= the collector origin, with no path.otlp_headersholdsx-api-key/x-secret-key, percent-encoded.project_nameandproject_type=observecome fromOTEL_RESOURCE_ATTRIBUTES, becauseinit()has no resource argument.Measured with the real
openlit1.45.0 andopenai2.54.0 against a loopback OpenAI fake and the shared harnessReceiver:/v1/traces.<origin>and<origin>/reach/v1/traces.<origin>/tracerreaches/tracer/v1/traces..../v1/tracesURL doubles the path and gets a 404.,and=. Without encoding, the secret is cut at the comma.gen_ai.tool.argsand callergen_ai.request.user(tested with planted markers); the Responses API tool-call path has the same issue by source reading, not tested.@openlit.tracerecords function args/return values independently./v1/metricsand/v1/logs, which fi-collector does not serve. The recipe disables both.init()downloads a price table fromraw.githubusercontent.com; it was the only outbound attempt observed. The recipe passes a local empty table, and the run made 0 non-loopback attempts.POSTspan from the httpx instrumentor that OpenLIT enables.model,provider,prompt_tokens,completion_tokensandobservation_type=llm, and derivestotal_tokens.gen_ai.usage.cost, andgen_ai.input/output.messagesas input/output. These are listed as SF-1 processor gaps.disabled_instrumentors=["openai"]leaves a single model span (tested).Read from source only (not tested):
register()already set a global provider,openlit.init()reuses it and adds no exporter.FirecrawlApp. Its content switch never reaches that instrumentor (parameter-name mismatch), so it records scraped content regardless. TH-8325 wraps the v2 client. README note only; neither package was changed.init()leaves the Future AGI keys inos.environ, so child processes inherit them.Deviations from the spec (approved: Nikhil 2026-10-03 blanket)
post_otlp()body: a realopenlit.init()run captured the inventory, so the inventory is measured, not pending. This is the same choice reviewers accepted on TH-8337.gen_ai.cost.*/llm.cost.*arrived. In fact, fi-collector prices from model and tokens with its own table when no user cost arrived (converter.go:285-294). OpenLIT'sgen_ai.usage.costis still not a cost alias (adapter.go:311-313). Collector pricing was not exercised.init()) sends no Future AGI keys (401) and noproject_name(400). The recipe adds both. The future-agi/docs page is not edited here.code: 401and the app exits 0.Review-r1 fixes / privacy caveats
Round 1 requested changes at 87cfaee (reviewer t_51e2fbbe, verifier t_0ff41c63): R1 P2 was an incomplete privacy claim, not a new vendor behavior. Fix round a8d3023:
Review round 2 (pr-verifier t_09054c97 at
a8d3023: VERIFIED)All seven round-1 findings resolved, no P1/P2. Retained P3 V3: the opt-in input/output drift check matches raw Go text, so a commented-out old assignment can keep it green after the real extraction changes (shown in the demo's drift chapter). Companion docs and architecture corrected to this head in company-brain #63 (
bdfcea9, open).Tests
Head
a8d3023. Coordinator rerun from the repo root on a clean tree (tickets/TH-8338/exact-head-a8d3023-checked/results.json):The skips are the 2 traceai-openai tests (they need
--with 'traceai-openai==0.1.10', which resolves wrapt 1.17.3) and the opt-in fi-collector alias check (FI_COLLECTOR_SRC= fi-collector at future-agimain4af5338).14 files, all under
python/examples/openlit/, with 0 secret-like added lines.What the tests cover:
Limits
observation_type, 400/401 and stamping are read from future-agimain4af5338; an opt-in test checks the README table against that source.OTEL_EXPORTER_OTLP_TRACES_ENDPOINTare source-only.wrapt<2throughfi-instrumentation-otel1.1.0. The repo source oftraceai-openairaises aTypeErrorat instrument time under wrapt 2.5.0. That was observed by the builder and is outside this PR.Video demo
Recorded build: head
a8d302349860257b2663f62a8968384c394cbc95(this PR); 10 real terminal chapters of the demo driver from a clean read-only worktree plus intro and review cards; 7:05, 12 chapters, narrated with captions.The video (SHA-256
3cc4516eeefa4b4b3f691c9ca916e9610e3ec4288e1193c4a88122f052796468), captions (SRT), transcript, chapter list, preview and both media-verification notes are private attachments on Linear issue TH-8338 (Future AGI workspace; sign-in required). They are not linked here as raw upload URLs.register()firstWhat it shows: one trace (chat > POST, gpt-4o-mini 11/7/18) to
/v1/traceswith header names only and no pricing download; capture off removes prompt/answer/system text but tool-call arguments, the caller'suserand provider error text still arrive (D-8338, no filter); a globalregister()takes over OpenLIT's spans; inherited trace-specific OTLP settings override the recipe; the retained V3 drift-check false positive.Independent media checks (Rick): full audio/video decode exit 0; 12 embedded chapters match the list; the only audio gaps are the 11 chapter transitions; every end-of-chapter frame inspected (header names only, no ports, paths or key values); 7 delivered files scanned with zero secret/path/port/header hits (the recorder's note is delivered path-redacted). Before recording, Rick fixed two wrong preconditions in the Codex-built driver (no printed text changed) and ran all chapters twice with identical output. Not shown: a live model, OpenLIT host, running fi-collector, storage, auth or the trace view.
Stacked on #203 (shared harness,
3eaadc8), basedev. Not merged.Linear: TH-8338. approved: Nikhil 2026-10-03 blanket
Verification sweep (2026-10-12)
dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).Re-verification (2026-10-16)
Head
5d94039(one new commit ona8d3023: README "Live check" section; no code or test change).Versions. Latest stable upstream = pinned:
openlit1.45.0 (PyPI latest 2026-10-16),openai2.54.0 (latest 2.x; openlit requires<3), Python 3.12,opentelemetry-sdk/exporter 1.45.1.Real E2E against local Future AGI (fi-collector :14338, project
th8103-rick-openlit), the recipe'sinit_tracing()unchanged, CLIProxygpt-6-luna: agent with two chat calls and a tool between them, plus one unknown-model failure.fi_verifyPASS:UI screenshot checked: trace tree, agent graph and attributes render.
Live vendor verification (2026-10-17 keys). One real OpenAI run (gpt-4o-mini, a few cents): trace
dc53a9fc-ff5c-e2c1-4b32-d29013c55d59(llm tokens 75/22 and 90/13, hierarchy as above) and failure tracef2140add-d198-5115-4265-1db8ab3d085e(NotFoundError, ERROR), projectth8103-rick-openlit-live.Privacy proof (all stored columns of the 8-row project scanned, unique markers). Content off (recipe): the planted prompt marker never appears in
input/output; the system-prompt text and question text in chat calls are absent. As the README already documents, these are still stored: model-generatedgen_ai.tool.args,gen_ai.request.user,@openlit.tracefunction args/return, and the provider error body instatus_message. Not stored: stack traces (the local collector kept only the twoexceptionevent names and timestamps), usernames, hostnames,/Users/paths, emails. Content on (capture_message_content=True): prompt/answer appear only inattrs_string(gen_ai.input/output.messages); theinput/outputcolumns stay empty (trace UI shows no I/O), confirming the documented SF-1 gap.gen_ai.usage.costis not kept in attrs on this stack;costcolumn 0.Tests. PR suite on Python 3.12 with the README command: 42 passed, 3 skipped (2 traceai-openai tests, 1 opt-in collector-source test), re-run after the README commit.
Changes and why. README only: replaced "No fi-collector was run" statements and added "Live check". No code fix was needed: Rick's PR was correct as-is.
Companion docs page: future-agi/docs#1011 (draft, base main, no navigation.ts). Suggested nav: Integrations > traceAI > "OpenLIT".
🤖 Generated with Claude Code