Repository navigation
feat(examples): LangSmith OTEL trace-forwarding recipe and contract test (TH-8335) - #231
Merged
Merged
Conversation
The Node OTLP exporter streams with Transfer-Encoding: chunked and sends no Content-Length. The Receiver read Content-Length only, so every Node export came back 400 with zero spans; two TH-8103 children had to put a de-chunking relay in front of it. Receiver now accepts both framings. It also records one entry per accepted export (path, lower-cased headers, flattened resource attributes) via requests(), so a contract test can assert X-Api-Key/X-Secret-Key and project_name through the shared harness instead of a private recorder. Verified: 6 harness tests pass, and the TanStack example's real Node exporter delivered 4 chunked exports (4 spans, collector path, both auth headers, project_name/project_type) with no relay. Refs: TH-8339, TH-8103
_guarded_run.py runs a script with every non-loopback connect, connect_ex and getaddrinfo refused and logged as JSON lines, so the contract test can show the LangSmith SDK never tries to reach api.smith.langchain.com. _guard_probe.py is the positive control: it tries TEST-NET-1, the IPv6 documentation prefix and a .invalid name and must see all three refused. Same guard as the OpenLLMetry recipe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8335
No traceAI package, no proxy, no LangSmith REST client. With LANGSMITH_TRACING and LangSmith's OTEL mode on, langsmith 0.14.4 turns each run into an OpenTelemetry span on the global tracer provider. init_tracing() installs that provider before the LangSmith client is created: project_name/project_type on the resource and a standard OTLP/HTTP exporter configured by OTEL_EXPORTER_OTLP_* variables. The provider is needed because LangSmith's own provider builds its resource without OTEL_RESOURCE_ATTRIBUTES (no project_name, which fi-collector requires) and, without OTEL_EXPORTER_OTLP_HEADERS, sends the LangSmith key as x-api-key to the OTLP endpoint. The app is a @Traceable chain run calling a tool run and an LLM run (a stand-in returning usage_metadata), then flushes the LangSmith client and shuts the provider down. requirements.txt pins langsmith[otel] 0.14.4 and the OpenTelemetry SDK/exporter 1.45.0 it was tested with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8335
Live forwarding only: no historical import, no REST client, no interceptor, proxy, DNS override or certificate in front of api.smith.langchain.com. States which half was tested (the SDK half with langsmith 0.14.4 in OTEL-only mode, no key, loopback only; the collector half against the harness receiver; the LangSmith side of fan-out not at all, since a key is a spend hard stop). Documents the two modes as read from the 0.14.4 source and seen in the tests: OTEL-only (LANGSMITH_OTEL_ONLY or LANGSMITH_TRACING_MODE=otel) makes no LangSmith call; the official two flags give hybrid mode, which also calls GET /info and POST /runs/multipart. Also the span and attribute shape, how fi-collector types each run type (retriever runs land as embedding), content and the hide switches (which also drop the token counts), exported env metadata, and the exact test commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8335
fixtures/langsmith-0.14.4-run-tree.otlp.json is an OTLP/JSON body written by hand from the langsmith 0.14.4 OTEL exporter source: a chain run with a tool child and an LLM child, project_name and project_type=observe on the resource. It is posted with post_otlp() and checked with compare(); the tests assert one trace, the tool and LLM children, and the model, provider and token keys. No LangSmith needed. With langsmith 0.14.4 installed, src/app.py runs under the loopback guard in OTEL-only mode with no key and an unset LANGSMITH_ENDPOINT; its export must equal the fixture. Further runs cover the request path, headers and resource, OTEL-only vs LANGSMITH_TRACING_MODE=otel vs hybrid against a loopback LangSmith stub, tracing off, the hide switches, the traces endpoint, and LangSmith's own provider (sdk_own_provider.py). The first test fails if a LangSmith key is in the environment. Two opt-in tests read fi-collector's span-type rules and alias lists from FI_COLLECTOR_SRC and check the README tables and the fixture's types and hot keys against them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> approved: Nikhil 2026-10-03 blanket Refs: TH-8335
…ssover Clear both trace-specific endpoint/header overrides in the shipped README setup before configuring Future AGI. Exercise those exact shell lines with the real SDK exporter and two loopback receivers for stale endpoint, headers and combined cases. Narrow SDK, socket-guard and metadata privacy claims. RED at ddc83e0: three behavioral failures, no collection errors. Working candidate: Python 3.10–3.13 default 25 passed/2 opt-in skips; collector-source 27 passed each, actual rc0/JUnit. Clean-head rerun follows. approved: Nikhil 2026-10-03 blanket Refs: TH-8335
…README re-verification note (TH-8335) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…API (TH-8335) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a LangSmith trace-forwarding recipe at
python/examples/langsmith/(Linear TH-8335, decision D-8335). It is not a package and contains no proxy, MITM or LangSmith REST client. An ordinary LangSmith@traceableapp sends its runs to Future AGI's fi-collector as OpenTelemetry spans, using LangSmith's own OTEL mode.src/app.py:init_tracing()installs a global OpenTelemetryTracerProviderwithproject_nameandproject_type=observeon the resource, plus a standardOTLPSpanExporter(). The exporter takes its endpoint and headers fromOTEL_EXPORTER_OTLP_*. LangSmith adopts that provider when the client is created after it.project_name, and fi-collector rejects a batch without it (400). It builds the resource with the plain constructor, soOTEL_RESOURCE_ATTRIBUTESis ignored (langsmith/_internal/otel/_otel_client.py:124-131). WithoutOTEL_EXPORTER_OTLP_HEADERS, it also sends the LangSmith key asx-api-keyto whatever OTLP endpoint is configured (:106-122).client.py:240-306):LANGSMITH_TRACING=trueplusLANGSMITH_TRACING_MODE=otel(or the legacyLANGSMITH_OTEL_ONLY=true) gives OTEL only. No LangSmith call is made (client.py:1918-1921;_background_thread.py:640-661).LANGSMITH_TRACING+LANGSMITH_OTEL_ENABLED) give hybrid: LangSmith REST and OTEL in the same process (client.py:304-305,_background_thread.py:455-485). That is fan-out without a collector.tests/fixtures/langsmith-0.14.4-run-tree.otlp.json: a hand-built OTLP/JSON body with a chain root and tool and LLM children. It uses the keys 0.14.4 actually writes (_otel_exporter.py:68-109):gen_ai.request.model,gen_ai.system,gen_ai.usage.*,gen_ai.prompt/gen_ai.completionandlangsmith.*.tests/test_langsmith_recipe.py: 27 collected cases (below). It also includes a loopback guard with a positive control, shared with the OpenLLMetry recipe.Which half is tested
post_otlp()and checked withcompare(). Two opt-in tests (FI_COLLECTOR_SRC) read fi-collector'sconverter.goandadapter.gosrc/app.pyruns as written in a subprocess with the real langsmith 0.14.4, OTEL-only. There is no LangSmith key,HOMEis empty,LANGSMITH_ENDPOINTis unset, and the guard intercepts Pythonsocket.connect,connect_exandgetaddrinfo(not every OS/native/UDP/DNS path)LANGSMITH_ENDPOINTon a 127.0.0.1 stubGET /infoandPOST /runs/multipartand also exports OTEL. The LangSmith side against LangSmith's real host is untested: a key is a spend hard stop and was not requestedWhat fi-collector does with these spans (source reading at future-agi
main4af5338)/v1/tracesand/tracer/v1/traces(pkg/server/server.go:233-234);X-Api-Key/X-Secret-Key(pkg/auth/middleware.go:45-46). A resource withoutproject_nameis a 400 (pkg/auth/stamp.go:31-46).gen_ai.operation.namedecides (exporter/clickhouse25exporter/converter.go:79-131): chain → chain,execute_tool→ tool,chat→ llm.langsmith.span.kindis not read. From LangSmith's own map, a retriever run becomesembeddings, which is mis-typed as embedding (recorded, no collector change). A parser run becomes unknown.gen_ai.request.model,gen_ai.systemandgen_ai.usage.*are on the alias lists (pkg/adapter/adapter.go:279-310), so model, provider and tokens fill on the LLM span. Chain and tool spans carrygen_ai.system=langchain.gen_ai.prompt/gen_ai.completionare single JSON strings, not the indexed keys the backend's OTel GenAI adapter turns into messages (futureagi/tracer/utils/adapters/otel_genai.py:56-57, :135-136). Rendering is not tested.Privacy
LANGSMITH_HIDE_INPUTS=trueandLANGSMITH_HIDE_OUTPUTS=trueremove the tested prompt/completion copies (tested), not metadata. The cost: token counts come from the outputs, so the token columns empty too (tested).LANGSMITH_*/LANGCHAIN_*env var, plusrevision_id(git describeof the working directory), is exported aslangsmith.metadata.*(env/_runtime_env.py:172-222). This filters names, not values: allowed endpoint/project values may contain credentials.LANGSMITH_HIDE_METADATAdrops the metadata, along with the model name. Raw inputextra_headers, query/body fields may survive ingen_ai.promptunless inputs are hidden.Deviations from the spec (approved: Nikhil 2026-10-03 blanket)
client.py:734-760, :1918-1921), so that half was run under the guard. A child process imports langsmith; the architecture said none would, and the brief allowed it once the source showed it was safe.LANGSMITH_OTEL_ONLYexists and works in 0.14.4 as a legacy name forLANGSMITH_TRACING_MODE=otel(client.py:302-303). The architecture said to mention it only once that was shown; the tests show it.llm.*aliases from the architecture's evidence list.@traceablerun types, not langchain-core runnables or LangGraph callback runs.TH-8335-docs.mdx) is out of date ("fixture-only", "does not set up forwarding"). Its amendment in company-brain Dev #63 is still pending; do not treat that original draft as a consistent published contract.Review repair — round 2
Previous head
ddc83e0b462c8d3c1eaf0b833bdeb6179caae866; new head4c843f2046df98d84e628abcd11ad464a9db6ae0. Delta is README plus tests only; app, fixture and shared harness are unchanged.OTEL_EXPORTER_OTLP_TRACES_ENDPOINTorOTEL_EXPORTER_OTLP_TRACES_HEADERScan override the copied FI general settings and cross destinations/credentials. The Run block now unsets both before the FI exports, retaining standard exporter behavior.export/unsetshell lines using dummy credentials, followed by the real SDK exporter and two loopback receivers. Cases: stale endpoint, stale headers, both. RED on old README: three behavioral assertion failures, actual rc1, no collection errors/skips. Do not mistake the worker's separate cache-denied precollection rc2 for behavioral evidence.t_f13e5c86VERIFIED the delta at4c843f2(no P1/P2): V1, R1, R2, R4, R5 resolved; R3 stays killed. Not a merge authorization._child_env, so deleting the README's two generalexportlines, or moving theunsetlines afterpython src/app.py, would not fail it. Proposed fix: pop both general variables before running the README lines and assert the unsets precede the run line._guarded_run.pyand the test module docstrings still describe full network isolation; the README is already corrected. Proposed fix: narrow both docstrings.LANGCHAIN_TRACINGfallback.Tests
Clean committed head
4c843f2046df98d84e628abcd11ad464a9db6ae0; actual exit codes and JUnit captured independently by Rick. LangSmith 0.14.4, OTel HTTP exporter/sdk 1.45.0, pytest 9.1.1. The two collector-source cases read the pinned source; they do not execute a collector.The older pytest-only result (8 passed/16 skipped at ddc83e0) is historical, not rerun at this head. Logs/results/JUnit are retained for the private Linear evidence handoff. The new regression cases are GREEN in the full supported matrix. No live LangSmith/collector calls, real account keys or paid model calls were made.
Limits
hide_*/anonymizercallables;connect,connect_exandgetaddrinfoonly, not OS/native/UDP or all DNS APIs. The pinned Python HTTP export path is exercised.Video demo
Recorded build: head
4c843f2046df98d84e628abcd11ad464a9db6ae0(this PR); 10 real terminal chapters of the demo driver from a clean read-only worktree plus intro and review cards; 7:00, 12 chapters, narrated with captions.The video (SHA-256
1f168ac142b0123cee5e44efe315179daa84e1462147b91f943df5fe242480a3), captions (SRT), transcript, chapter list, preview and both media-verification notes are private attachments on Linear issue TH-8335 (Future AGI workspace; sign-in required). They are not linked here as raw upload URLs.init_tracingexistsWhat it shows: the README path exports 3 spans (chain > tool, llm; gpt-4o-mini 21/9/30) with header names only and the project resource; the old README setup sends traces or keys to the wrong receiver in 3 of 3 stale-variable cases and the new one in 0 of 3; hybrid mode calls a loopback LangSmith stub as well as OTLP (the real LangSmith service is untested); a
LANGSMITH_TRACING_MODE=otelrun differs from the fixture only in LangSmith's switch-name metadata keys.Independent media checks (Rick): full audio/video decode exit 0; 12 embedded chapters match the list; the only audio gaps are 11 chapter transitions (about 1.7 s) and two disclosed reading holds (2.3 s and 2.9 s); every end-of-chapter frame inspected (loopback shown without a port, header names only); 7 delivered files scanned with zero secret/path/port/header hits (the recorder's note is delivered path-redacted). Chapter 11 reads the verify-r2 follow-up list as recorded, so it still lists R6 (companion docs) as pending; R6 was completed afterwards in company-brain #63 at
886fe7d, as chapter 12 says. Not shown: real LangSmith, a running fi-collector, storage, auth or the trace view.Stacked on #203 (shared harness,
3eaadc8), basedev. Not merged.Linear: TH-8335. approved: Nikhil 2026-10-03 blanket
Verification sweep (2026-10-12)
dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).Re-verification (2026-10-16)
init_tracing()and the README setup work unchanged on both.python/examples/langsmith/e2e/real_langchain_run.py(new, opt-in) runs a real LangChain agent (gpt-5.6-terra via the local proxy) with one tool, then a run whose tool raises. Projectth8103-rick-langsmith-trace-forwarding, traces10876845-8034-8f99-c129-55ea117a39cf(ok) and77f22692-106e-794c-97bb-08ecdc39822d(failure); pinned leg in projectth8103-rick-langsmith-pin.chain LangGraph > chain model > llm ChatOpenAI model=gpt-5.6-terra tok=357/19,chain tools > tool lookup_policy; failure tracetoolsandbroken_lookuphave status ERROR (no error text stored). Same shape on 0.14.4.gen_ai.prompt/gen_ai.completionattributes (visible under Attributes). Not fixed here; it is a collector/backend mapping item (this PR does not touch them). Documented on the new docs page.LANGSMITH_HIDE_INPUTS=true LANGSMITH_HIDE_OUTPUTS=true: 0 of 7. Local username,/Users/paths and hostname: 0 in both runs. Tool error text and tracebacks: not stored. Known:langsmith.metadata.revision_idisgit describeoutput (tag or commit hash), and token counts empty when outputs are hidden.LANGSMITH_TEST_LATEST=1(only the pin-equality test is skipped; the fixture export comparison passes on 0.14.11).e2e/real_langchain_run.py, aLANGSMITH_TEST_LATESTleg in the pin test, README note. Rick's recipe code was correct as-is.src/pages/docs/integrations/langsmith.mdx(no navigation.ts edit). Suggested nav: Integrations > Frameworks, "LangSmith".Live vendor verification (2026-10-19)
Fan-out mode against the real LangSmith API (US,
api.smith.langchain.com, personal access token), spans to a local Future AGI stack. LangSmith projectth8103-e2e(deleted afterwards), Future AGI projectth8103-livefill-th8335. Versions: langsmith 0.14.11, langchain 1.4.6, langchain-openai 1.7.2, opentelemetry-sdk/exporter 1.45.1, Python 3.12.GET /info200,POST /runs/multipart202);POST /api/v1/runs/queryreturned the full run tree with token totals.01a15682-3f4a-77ae-8845-0786985be886= Future AGI trace40aac0f3-d1cf-e174-f734-f7e572573276(7 spans; llmgpt-5.6-terra357/19 and 398/18 tokens; toollookup_policy).fi_verifyPASS.01a1568c-9cc1-7217-8cfb-0078e9343cfb(error) = Future AGI tracef8033fac-dd16-c3e0-6ada-af81b58d49af(5 spans,broken_lookupandtoolsin ERROR). LangSmith keeps error text and traceback; Future AGI stores neither.lc_run--<run id>text in the content).LANGSMITH_HIDE_INPUTS/OUTPUTS=true(all ClickHouse columns scanned); the hide switches also nullinputs/outputson the LangSmith copy (read back via API). Username, host name,/Users/,/private/,@, "Traceback" and tool error text: 0 hits in all 46 scanned rows across the three projects.OTEL_EXPORTER_OTLP_TIMEOUT). No recipe code change was needed.LANGSMITH_TEST_LATEST=1on 0.14.11. Docsdocs_build.sh: build and audit-links pass.🤖 Generated with Claude Code