Repository navigation
fix(langchain): Deep Agents compatibility, cookbook and thread_id as session.id (TH-8245) - #214
Merged
Merged
Conversation
Deep Agents (deepagents 0.7.21) is a LangGraph graph, so per D-8245 there
is no new package: a cookbook plus a compatibility test on the existing
LangChainInstrumentor. Validation found one defect, fixed here in
traceai_langchain.
Fix: LangGraph copies configurable.thread_id into run metadata, but the
tracer only emitted it as thread_id / gen_ai.conversation.id, never as
session.id, so a run passing config={"configurable": {"thread_id": ...}}
had no session unless wrapped in using_session(). _update_span now falls
back to thread_id for session.id. using_session() and
metadata["session_id"] still win. Unit tests: test_session_from_thread_id.py
(3 of 6 fail without the fix; the precedence cases pass either way).
Added:
- examples/deep_agents.py: create_deep_agent with an explicit scripted
fake model (usage_metadata on every turn), one custom tool, StateBackend,
register() + instrument() with hide_inputs/hide_outputs, thread_id t-1,
force_flush().
- tests/test_deepagents_compat.py: skips without deepagents. AC-01..AC-05
in memory, a privacy control pair, and a shared-harness contract test
that runs the cookbook as a subprocess through the real register()
exporter into python/tests/harness Receiver.
- README Deep Agents section with the versions the test printed; CHANGELOG.
Findings (deepagents 0.7.21, langchain-core 1.6.6, langgraph 1.2.12):
- session.id: missing before this fix (see above); now t-1 on every span,
subagent spans included, also through the real exporter.
- Parenting: the general-purpose subagent span (CHAIN) is a direct child of
the task TOOL span; its 2 LLM spans and ls TOOL span are below it; one
trace id. task is not wrapped.
- Double counting: none. Token keys sit only on LLM spans; trace-wide sum
of gen_ai.usage.input_tokens = 143 = sum of the 7 model calls.
traceai-langchain emits gen_ai.usage.*, not llm.token_count.*.
- write_todos: not bound by default (TodoListMiddleware is added only by
the OpenAI Codex harness profile, profiles/harness/_openai_codex.py).
- Span kinds: middleware nodes named *before_agent are AGENT through the
existing name heuristic; graph root, subagent root and nodes are CHAIN.
Recorded, not changed.
- Errors: a raising tool ends TOOL, tools node and root ERROR with an
exception event; langgraph's default ToolNode handler re-raises, so
invoke() raises. Cancelled astream leaves no open span; the root ends
ERROR with a GeneratorExit description. Recorded, not changed.
- No gen_ai.provider.name; LangChain's raw ls_provider passes through.
Deviations from the TH-8245 spec, with evidence:
- Tool list: 0.7.21 also binds `delete` (filesystem.py _FS_TOOL_ORDER),
absent from the graph.py docstring the spec copied. `execute` is not bound
at all on a non-sandbox backend (_unsupported_tools), rather than bound
and erroring. Evidence: tool names captured from bind_tools in the test.
- Pin: the PyPI 0.7.21 wheel is not byte-identical to f57c6f38 (graph.py
and middleware/subagents.py differ in skills-middleware code;
filesystem.py, _version.py and the tool docstring match). Tests ran on
the wheel; README names the wheel, not the SHA.
- langchain-core matrix collapses to one point: 1.6.6 is both deepagents'
floor and the latest on PyPI (checked 2026-10-04).
- Test env needs wrapt<2: wrapt 2.5.0 makes instrument() raise TypeError
(wrap_function_wrapper keyword `module`). fi-instrumentation-otel 1.1.0
declares wrapt>=1.15,<2, so pip installs are unaffected.
AC-08: existing suite unchanged and green at langchain-core 1.6.6 and at
the 0.2.43 floor (langgraph 0.2.76); lower bound not raised.
approved: Nikhil 2026-10-03 blanket
Refs: TH-8245
…precedence, blank thread_id R1: README, cookbook and version table no longer present the thread_id -> session.id fallback as part of the published 0.2.0 (same version string); 0.2.0 users are told to wrap the call in using_session(thread_id). R2: documented precedence is the real one: metadata session_id > using_session > thread_id (pre-existing order), with a test. R3: the per-run fallback is documented (set the session around the whole invoke); a test pins a session set inside a node. R4: examples/requirements.txt and README say deep_agents.py needs its own environment and that the compat test runs manually, not in CI. R5: write_todos qualified (harness profiles can add it). R6: the cancelled-astream root ERROR/GeneratorExit is now asserted. R7: a blank thread_id sets no session.id (test failed first: 2 cases). R8: langgraph-prebuilt added to the version table and the version print. approved: Nikhil 2026-10-03 blanket Refs: TH-8245
Token counts came from the provider's raw llm_output usage first and LangChain's usage_metadata only when that was missing. Anthropic's raw input_tokens leaves out cached prompt tokens. Deep Agents turns on Anthropic prompt caching by default, so a real Deep Agents run on ChatAnthropic showed 4 input tokens for a 4018-token prompt on every LLM span, and the trace total was off by three orders of magnitude. usage_metadata (LangChain's normalized usage, which counts cached tokens as the GenAI conventions do) now comes first; the raw usage is the fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_tested_versions asserted deepagents == 0.7.21, so the compatibility suite failed on the current release (0.7.23) although every other test passes there. Assert >= 0.7.21 instead, and list the newer versions that also pass in the README's tested-versions table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s/outputs are hidden (TH-8245) run.error is repr(error) plus the formatted traceback, and it was copied into the span status description; record_exception also added exception.stacktrace. Both carry local file paths and usernames. Status description and exception event now hold the exception type and message only, and only the type when TraceConfig hides inputs or outputs. Re-verified on deepagents 0.7.28 and 0.7.21. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
9 of 10 tasks
…ace (TH-8245) test_record_exception still expected Span.record_exception, which this branch stopped calling because it adds exception.stacktrace. The test now checks the hand-built exception event (type and message, no stack trace) and that hide_error_text keeps only the type. The package is on the known-broken list, so CI would have hidden this third failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
traceAI-langchain 0.2.0 is on PyPI with different code, so the thread_id session, token count and error text fixes ship as 0.2.1. README and the Deep Agents example now name 0.2.1 instead of 'unreleased'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retain session mapping and token precedence; preserve core error masking and explicit false with a published-core privacy fallback. Add nested error/session regressions and Deep Agents CI. Co-Authored-By: Claude Code <noreply@anthropic.com>
Defer new TraceConfig construction so published core collects the full suite. Test legacy derived privacy and native error policy explicitly. Replace obsolete helper mocks with actual span assertions. Remove LangChain from the known-broken allowlist and run all package/core helper tests against both core versions in exact-head CI. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TH-8245 checks that the existing
traceAI-langchaininstrumentor traces LangChain Deep Agents (deepagents0.7.21), and adds a runnable cookbook. One defect turned up and is fixed here:thread_idwas never copied tosession.id.traceai_langchain/_tracer.py):session.idnow falls back to the run'sthread_id.metadata["session_id"]andusing_session()still win, in that order (pre-existing). A blankthread_idsets nothing. It is set on every span, subagents included, and also through the real exporter.using_session(thread_id)on 0.2.0).python/frameworks/langchain/examples/deep_agents.py: a Deep Agent with the built-in filesystem tools and ageneral-purposesubagent throughtask, traced withregister()+LangChainInstrumentor.What the test confirms on 0.7.21:
taskTOOL span, with its LLM and tool spans under it.taskis not wrapped.gen_ai.usage.*is on LLM spans only, and the trace-wide input-token sum (143) equals the 7 model calls. Nollm.token_count.*keys are emitted.bind_tools.Deviations from the accepted spec (with evidence)
delete(_FS_TOOL_ORDERinfilesystem.py), which thegraph.pydocstring the spec copied leaves out.executeis not bound on the default in-memory backend.write_todosis not bound by default at 0.7.21; only the OpenAI Codex harness profile adds it.f57c6f38(skills-middleware code ingraph.pyandmiddleware/subagents.py). Tests ran on the wheel, and the README names the wheel.langchain-core>=0.2.43) is unchanged.Tests (head 46de0af, clean tree)
Rerun by the coordinator:
python/frameworks/langchain/testsThe skip is
test_dependency_pins.py:90, which needsFI_RUN_SLOW=1.Command:
Implementer runs:
session.idassertion.The tests need
wrapt<2.fi-instrumentation-otel1.1.0 already declareswrapt>=1.15,<2; a fresh resolve oftraceAI-langchainpicks wrapt 1.17.3, so pip installs are unaffected.Review
Round 1, run as separate installed cards on Opus 5.5:
All are fixed in 46de0af. The R7 test failed first (2 cases).
using_session(thread_id)session_id>using_session>thread_idexamples/requirements.txtand README saydeep_agents.pyneeds its own environment and the compat test runs manuallywrite_todosclaim unqualifiedthread_idgavesession.id=''langgraph-prebuiltmissing from the version table/printRound 2 (pr-verifier t_9126508b on 7bc464f..46de0af): VERIFIED. R1-R8 resolved, no regressions, no blocking findings. Review is complete at 46de0af.
P3 follow-ups:
session.idis unqualifiedlanggraph-prebuiltversion (1.1.0)examples/requirements.txtunderstates the langgraph floor that langchain 1.4.3 needsVideo demo
A narrated before/after terminal demo (4:31) was recorded at the reviewed head
46de0af2f09326da3f9e389898bd76efd3bd5f06, against base3eaadc8. It is a real terminal recording, not a slideshow.What it runs:
register()exporter into the shared harness Receiver.Where to find it: the video, captions, transcript and media check are attached privately to Linear issue TH-8245 as
th-8245-demo.mp4,th-8245-demo.srt,transcript.mdandmedia-verification.md. The mp4's sha256 isd99fb612…bed71b79.Chapters:
session.idNone on all 28session.id=t-1on all 28, subagent included_session_id_from_thread_id, precedence, blank guard, per-run choice, unreleased noteLimits
*before_agentget kind AGENT from the existing name rule. Cancelling anastreamleaves no open span, but the root ends ERROR with aGeneratorExitmessage. Nogen_ai.provider.nameis set; only LangChain'sls_providerpasses through. All three are recorded, not changed.Stacked on #203 (shared harness). Not merged.
Linear: TH-8245. approved: Nikhil 2026-10-03 blanket
Verification sweep (2026-10-12)
dev, and the spans were checked in ClickHouse and in the Observe UI (hierarchy, span types, input/output, model and tokens, sessions, error status, and a secret scan of everything stored).Re-verification (2026-10-16)
Head after this pass:
0363d4b(one new commit on top ofa406cfb, no force-push).Versions. Latest stable on 2026-10-16:
deepagents0.7.28,langchain1.4.6,langchain-core1.6.12,langgraph1.2.16,langgraph-prebuilt1.1.1,langchain-openai1.7.2,fi-instrumentation-otel1.1.0, Python 3.11.12 and 3.13.15. Pinned leg:deepagents0.7.21 (same dependencies).traceAI-langchain= this branch (the published 0.2.0 does not have thesession.idchange).Tests (same command as above, deepagents unpinned = 0.7.28): Python 3.11 78 passed, 1 skipped; Python 3.13 78 passed, 1 skipped; deepagents 0.7.21 on Python 3.11 78 passed, 1 skipped; old floor (langchain-core 0.2.43, langgraph 0.2.76, no deepagents) 65 passed, 2 skipped. The new
tests/test_error_text_privacy.pyfails 6 of 6 without the fix.Real E2E against a local Future AGI stack built from
dev, modelgpt-6-lunathrough an OpenAI-compatible proxy (ChatOpenAI),StateBackend, onewrite_file, one custom tool, onetasksubagent that callsls; instrumented exactly as the README shows (register()+LangChainInstrumentor().instrument()); spans checked in ClickHouse withfi_verifyand by eye in the Observe UI.8d883b6c-8a45-46a2-8337-b1139b9bd1d3, 25 spans, PASS. Hierarchy: root LangGraph chain,modelandtoolschains,ChatOpenAILLM spans withgpt-6-lunaand tokens (2367/53, 2436/28, 2479/61, 2563/53 main; 2014/29, 2059/17 subagent), TOOL spanswrite_file,lookup_weather,task(parent of thegeneral-purposesubagent chain, whoselsTOOL span and 2 LLM spans are nested under it), onesession.id(t-060e4b) on every span.74d1d151-a6d3-4c7b-9d20-09cdf178c37c, 13 spans, PASS (write_file,taskwith nested subagent LLM spans).62629915-b89d-4459-a2ee-3f1b5ccba6fe, 10 spans, PASS;lookup_weatherTOOL, thetoolsnode and the root allERROR.Privacy proof (unique marker written into the prompt, the file and the tool error; every column of the
spanstable scanned, includinginput,output,attrs_string,attributes_extra,resource_attrs,span_events,status_message):FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true), 25-span run5d8f7706-ad83-40e2-9415-ea17b690e330: marker absent from all columns; inputs/outputs are the 12-byte redaction value.8d883b6c-...: marker present ininput,output,attrs_string,attributes_extra.Traceback,/Users/, username, hostname,site-packagesorFile "anywhere in either run.Defect found and fixed in
0363d4b. On a failed run the span status description held LangChain'srun.error, which isrepr(error)plus the full formatted traceback, andSpan.record_exceptionaddedexception.stacktrace. Before the fix the failing-run scan (trace22932c1c-c810-444f-ba88-62930041372f, hide flags on) found the raw error text including the marker,Traceback,/Users/<name>/...andsite-packagespaths instatus_messageon 3 spans. After: status description andexceptionevent carry the exception type and message only, never a stack trace; withhide_inputsorhide_outputsonly the exception type name remains (ERROR:RuntimeError). Rescan of the fixed runs (62629915-...content on,60292506-7d5e-452f-b393-e0955c694b0bhidden): hidden run has no marker in any column, neither has any traceback or path. Files:traceai_langchain/_tracer.py, newtests/test_error_text_privacy.py, README and CHANGELOG. The cancelled-astreamdescription is nowGeneratorExit()instead ofGeneratorExit()Traceback ....Docs page (future-agi/docs, draft, base
main, nonavigation.ts):src/pages/docs/integrations/traceai/langchain-deep-agents.mdx; suggested nav group "Integrations > traceAI" (next to LangGraph), title "LangChain Deep Agents". PR link: future-agi/docs#1000.Unchanged / still open. Rick's deviations from the spec (no
write_todos,deleteis bound, wheel differs from the git SHA) hold at 0.7.28 (bound tools:ls,read_file,write_file,edit_file,delete,glob,grep,task). No CI job installsdeepagents. Cancelling anastreamstill ends the rootERROR. The PR is not a draft (left as is).🤖 Generated with Claude Code