Skip to content

Add manual distributed tracing propagation for LLM Observability - #12402

Draft
ncybul wants to merge 1 commit into
masterfrom
llmobs/distributed-tracing
Draft

Add manual distributed tracing propagation for LLM Observability#12402
ncybul wants to merge 1 commit into
masterfrom
llmobs/distributed-tracing

Conversation

@ncybul

@ncybul ncybul commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an explicit, manual API for propagating LLM Observability context across process boundaries that automatic instrumentation doesn't cover — e.g. an SQS worker reading its own message attributes. Mirrors dd-trace-py's inject_distributed_headers/activate_distributed_headers model (Python/Node/Go already support this; Java did not).

// Producer side
Map<String, String> headers = new HashMap<>();
LLMObs.injectDistributedHeaders(span, headers);
// ... put headers into SQS message attributes ...

// Consumer side
try (Closeable scope = LLMObs.activateDistributedHeaders(headers)) {
  // spans started here join the producer's trace and inherit session_id / agent attribution
}

Design

  • Explicit/manual, not automatic instrumentation. The customer calls this per message in their own consumer loop. This sidesteps a pre-existing TracingIterator batch-context-reuse bug in APM's automatic SQS instrumentation rather than fixing it — that's a separate, not-yet-filed issue.
  • Wire format: standard APM trace context (trace id, parent id, sampling) is injected/extracted via the normal Propagators.defaultPropagator(). LLMObs-specific tags (_dd.p.llmobs_ml_app, _dd.p.llmobs_sid, _dd.p.llmobs_pagent_span_id, _dd.p.llmobs_pagent_name) are appended as extra key=value pairs onto the same x-datadog-tags carrier entry the default propagator already writes, rather than registered as a new Concern/Propagator or added to PropagationTags/PTagsFactory in dd-trace-core. This keeps the change self-contained to agent-llmobs and preserves wire compatibility with dd-trace-py/js/go, which nest the same _dd.p.llmobs_* tags inside x-datadog-tags/tracestate t.*.
  • API surface: LLMObs.injectDistributedHeaders/activateDistributedHeaders (dd-trace-api), delegating to a new LLMObs.LLMObsPropagator interface, following the same static-field delegation pattern as SPAN_FACTORY/EVAL_PROCESSOR/FEEDBACK_PROCESSOR. Real implementation is DDLLMObsPropagator (agent-llmobs), wired in via LLMObsInternal.setPropagator(...); defaults to a no-op when the agent isn't active.

Known scope limitations

  • ml_app rides on the wire for cross-SDK compatibility but is not yet auto-applied to spans started after activateDistributedHeaders — callers must still pass ml_app explicitly when starting a span.
  • agent_version is not propagated cross-process (no SDK sends it today).
  • No cross-language system test yet — this is a Java-only manual API, not a shared automatic-instrumentation contract, so it's scoped as a fast-follow.

Test plan

  • New unit tests: DDLLMObsPropagatorTest (JUnit 5) — round-trips a plain Map<String, String> carrier (the shape an SQS message-attribute map would take): trace-id join across inject/activate, session_id/agent-attribution propagation, no-op on missing trace context, null-arg validation, ml_app always present even without session/attribution.
  • :dd-java-agent:agent-llmobs:test passes (no regressions).
  • spotlessCheck passes.
  • Self-reviewed against the tracer perf rubric — no hot-path findings (this API is explicitly/manually invoked at a message boundary, not on the automatic span/advice hot path).

Claude session: 15543c2c-2abe-408e-b16e-05ddbe972287
Resume: claude --resume 15543c2c-2abe-408e-b16e-05ddbe972287

Adds LLMObs.injectDistributedHeaders/activateDistributedHeaders so
applications can manually propagate LLMObs context (ml_app, session_id,
agent attribution) across boundaries automatic instrumentation doesn't
cover, e.g. an SQS worker reading its own message attributes. Standard
APM trace context rides the normal Propagators.defaultPropagator();
LLMObs tags piggyback on the existing x-datadog-tags carrier entry to
stay wire-compatible with dd-trace-py/js/go.

Claude session: `15543c2c-2abe-408e-b16e-05ddbe972287`
Resume: `claude --resume 15543c2c-2abe-408e-b16e-05ddbe972287`
@ncybul ncybul added type: feature Enhancements and improvements comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM labels Sep 3, 2026
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 74.03%
Overall Coverage: 57.45% (-1.56%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ce20e17 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.91 s 14.63 s [+0.9%; +2.9%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.63 s 13.71 s [-1.3%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 17.59 s 17.31 s [+0.7%; +2.5%] (maybe worse)
startup:petclinic:iast:Agent 17.39 s 16.93 s [-1.6%; +7.0%] (no difference)
startup:petclinic:profiling:Agent 17.61 s 17.41 s [-0.3%; +2.6%] (no difference)
startup:petclinic:sca:Agent 17.56 s 17.41 s [-0.2%; +2.0%] (no difference)
startup:petclinic:tracing:Agent 16.48 s 16.64 s [-2.0%; +0.0%] (no difference)

Commit: ce20e177 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant