Add LangGraph durable-execution agent lifecycle events - #536
Draft
meshailabs wants to merge 8 commits into
Draft
Conversation
Prototype producer for the candidate conventions in open-telemetry/semantic-conventions-genai#445, captured from unmodified LangGraph applications at two real interception points. LangGraph 1.2 dispatches on_interrupt and on_resume to the handlers in a run's callback manager, and the instrumentation's handler is already in that list, so gen_ai.agent.paused and gen_ai.agent.resumed come straight from GraphInterruptEvent and GraphResumeEvent with the interrupt ids and checkpoint ids LangGraph produced. Adding these two methods also removes the AttributeError LangGraph currently logs on every interrupt. The checkpointer passed to StateGraph.compile is wrapped so every persisted checkpoint emits gen_ai.agent.checkpointed, correlated to the live run through the LangGraph thread id. BaseCheckpointSaver.put is abstract and every saver overrides it, so the instance is patched rather than the base class. gen_ai.agent.execution.id and gen_ai.agent.pause.reason are omitted: LangGraph reports nothing with those semantics. See docs/design-notes/langgraph-lifecycle-events.md and the captured telemetry sample beside it. Claude-Session: https://claude.ai/code/session_01FAnWkWBL3mJ81AMv7KA6br
Review follow-up on the lifecycle events. Correlation now tracks a stack of live runs per LangGraph thread id, so a nested run never displaces the run containing it and the innermost live run owns the checkpoint. Two unrelated live runs on one thread id are ambiguous, so the event is dropped with a debug log rather than attributed to a guess. One lock now guards both the thread stacks and the invocation states, making a checkpoint lookup atomic against a concurrent chain end. Thread bindings prune dead runs and are bounded, so runs that never report an end cannot accumulate. Checkpointer patching checks that a saver is hashable and weak referenceable before touching it, so a saver that cannot be tracked is never left partly patched. The instance attributes replaced by the patch are saved and restored on uninstrument instead of deleted. Checkpoint de-duplication now remembers the last id per namespace rather than relying on context propagation, which covers savers whose aput delegates to put on a worker thread. The saver is discovered from the compiled graph rather than from the compile arguments. GenAIInvocation.emit_event is byte-identical to the hunk in PR open-telemetry#507; the design note and changelog now say so and that it is dropped when rebasing onto that PR. Claude-Session: https://claude.ai/code/session_01FAnWkWBL3mJ81AMv7KA6br
…identity Second review follow-up. Correlation now carries the LangGraph checkpoint namespace end to end. Each graph run binds the namespace its own checkpoints are written under, which is "" for a top level graph and the run's langgraph_checkpoint_ns for a nested graph. A write resolves to the run bound with that exact namespace, or to the nearest enclosing run when none is bound, and stays ambiguous only when two equally specific live runs are unrelated. LangGraph classifies a subgraph's own graph run as a plain chain rather than a workflow, so today a child namespace resolves to the parent workflow; the resolution is namespace aware regardless. Checkpoint de-duplication no longer remembers the last id per namespace, which could drop an interleaved delegated write or a legitimate re-persist of the same id. The identity of the Checkpoint object is held while a write is in flight instead: a nested call receiving the same object is the delegation and stays silent, and the outermost call reports. A saver that copies the checkpoint before delegating is reported twice, documented as a known limit. Adds tests for namespace resolution, nested resume provenance, interleaved and concurrent delegated writes, a separate write reusing a checkpoint id, and thread unbinding on chain end and chain error. The emit_event hunk from PR open-telemetry#507 is saved at docs/design-notes/pr507-emit_event.patch so its identity with the local change is checkable on disk. Claude-Session: https://claude.ai/code/session_01FAnWkWBL3mJ81AMv7KA6br
Third review follow-up. The interleaving test only synchronized thread entry, so the ordering it claimed to cover was never forced. A sequenced saver now blocks on per-call events so the test drives the exact sequence: outer A enters, the delegated A enters and returns, an independent outer B enters, outer A returns, outer B returns. It asserts two writes and exactly two events, one per write, proving the delegation is silent and neither independent write is dropped as a false duplicate. Verified in langgraph 1.2.9 that two independent writes can never share a checkpoint object: the loop hands the saver copy_checkpoint(self.checkpoint), a freshly constructed mapping, once per superstep. The design note records that assumption as a stated limit, along with the over-reporting fallback for a saver that copies before delegating, and its stale description of the previous id based de-duplication is rewritten. The PR open-telemetry#507 reference is now stored as extracted source text rather than a raw unified diff, so it carries no trailing whitespace, and a test checks the local emit_event against it. Claude-Session: https://claude.ai/code/session_01FAnWkWBL3mJ81AMv7KA6br
The design note, its generating script, and the captured telemetry lived under a new top-level docs/design-notes directory. They now live where the package's other runnable material does. The script becomes examples/langgraph-lifecycle/main.py alongside a requirements.txt and a README.rst, matching the shape of the sibling examples. The design note is now the README's design note section in reStructuredText, unchanged in substance, and the captured telemetry is sample-output.json referenced from it. The PR open-telemetry#507 reference source moves to util/opentelemetry-util-genai/tests/fixtures, next to the test that checks it, which now resolves the path from the test file rather than the repository root. The top-level docs/design-notes directory is gone and nothing else under docs/ changed. Claude-Session: https://claude.ai/code/session_01FAnWkWBL3mJ81AMv7KA6br
Author
|
Tracking in #537 |
Pull request dashboard statusWaiting on the author · refreshed 2026-09-02 14:52 UTC Move out of draft to request review. Status above doesn't look right?
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Emit gen_ai.agent.paused, gen_ai.agent.checkpointed, and gen_ai.agent.resumed from the LangChain instrumentation for LangGraph durable runs. These follow candidate OTel GenAI conventions (open-telemetry/semantic-conventions-genai#445), reset to draft pending real instrumentation evidence.
Hook LangGraph 1.2 on_interrupt/on_resume on the existing handler. LangGraph already dispatches them; today that path logs an AttributeError on every interrupt and resume. Report checkpoints by wrapping put/aput on the compiled graph's saver.
Known gaps: no gen_ai.agent.execution.id (LangGraph has no id that spans suspend/resume); pause.reason is not derivable from interrupt(value); resumed_from.type is always checkpoint; subgraph runs are chains, so child-namespace checkpoints land on the parent workflow span. emit_event in util-genai mirrors #507 and should drop on rebase if that lands first.
Type of change
How has this been tested?
tox latest and oldest matrices for the langchain instrumentation, the util-genai suite, typecheck, and pre-commit. Runnable example with a captured run under examples/langgraph-lifecycle/.
Checklist