Send Java SDK identity with Feature Flagging EVP events by leoromanovsky · Pull Request #12575 · DataDog/dd-trace-java

leoromanovsky · 2026-09-19T03:04:52Z

Motivation

Attribute Feature Flagging exposures and flag evaluations to the Java SDK on both local EVP proxy and direct-intake routes. Initial direct delivery already shipped; this PR isolates producer identity from the transport hardening in #12477. Tracks FFL-3016 within the FFLSDK-187 rollout.

Changes

Send DD-EVP-ORIGIN: dd-trace-java and DD-EVP-ORIGIN-VERSION using the tracer version.
Add immutable per-request header configuration to the shared backend factory.
Verify local/direct identity headers while retaining credential-isolation and one-send assertions.

Decisions

Stack on leoromanovsky/fflsdk-187-java-evp-fallback, not another language or unrelated SDK branch.
Forwarding-capability requirements, routing, retries, credentials and lifecycle remain in Harden Agentless Feature Flags EVP delivery #12477. Required header names are independent of the identity values emitted here.
No new configuration or system-test changes.

Validation

Exact base: 1b5766b56c67bb7151a084abb752e62e5bb4b332.
Exact candidate: 96a3427bb9bcb640165306d16af141e9455e2cc4.
./gradlew :communication:spotlessApply :products:feature-flagging:feature-flagging-lib:spotlessApply :communication:test :products:feature-flagging:feature-flagging-lib:test --console=plain --max-workers=4: PASS; communication 235/235, feature-flagging-lib 279/279.
git diff --check: PASS. New commit is GitHub verified: verified=true, reason=valid.
No current-candidate system-tests, dogfooding or backend-intake run. Older artifact results are not claimed for this split.
Fresh CI remains separate from local validation. This PR remains a draft.

Add capability-gated local discovery, safe sticky direct fallback, shared route state, send-once semantics, and bounded lifecycle handling for Java Feature Flags telemetry. Environment: Datadog workspace

Move identity attribution to a follow-up branch while keeping forwarding-capability requirements explicit and independent of emitted headers. Preserve credential isolation, no-replay coverage, and existing lifecycle hardening. Validation: communication tests 234/234 and feature-flagging-lib tests 279/279 pass; scoped Spotless formatting passes. Environment: Datadog workspace

Add immutable per-request identity headers independently of forwarding capability checks and send-once policy. Emit dd-trace-java and the tracer version for both exposure and flag-evaluation requests. Validation: communication tests 235/235 and feature-flagging-lib tests 279/279 pass; scoped Spotless formatting passes. Local/direct wire assertions preserve credential isolation and one-send behavior. Environment: Datadog workspace

linear-code · 2026-09-19T03:05:05Z

FFLSDK-187

datadog-datadog-prod-us1 · 2026-09-19T03:19:40Z

🎯 Code Coverage (details)
• Patch Coverage: 30.77%
• Overall Coverage: 61.51% (+1.65%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: ea60b85 | Docs | Give us feedback!}

dd-octo-sts · 2026-09-19T03:29:04Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.69 s	14.61 s	[-0.2%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent	13.50 s	13.64 s	[-1.7%; -0.3%] (maybe better)
startup:petclinic:appsec:Agent	17.13 s	16.87 s	[+0.6%; +2.4%] (maybe worse)
startup:petclinic:iast:Agent	16.98 s	17.01 s	[-1.0%; +0.6%] (no difference)
startup:petclinic:profiling:Agent	16.74 s	16.60 s	[-0.4%; +2.0%] (no difference)
startup:petclinic:sca:Agent	17.02 s	16.94 s	[-0.3%; +1.3%] (no difference)
startup:petclinic:tracing:Agent	16.02 s	16.05 s	[-1.2%; +0.8%] (no difference)

Commit: ea60b85f · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Environment: Datadog workspace

Gate shutdown instrumentation on tracing or resolved feature flag enablement, and share Agent path joining. Pin recovery generations, discovery concurrency, replay, idle recovery, and disabled-product behavior with regression tests. Environment: Datadog workspace

chatgpt-codex-connector · 2026-10-07T15:25:30Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T15:29:22.276016Z	`96a3427`	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-07T15:29:50.948765Z	`96a3427`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-datadog-prod-us1

More details

Java SDK identity headers reach both local EVP and direct-intake requests while preserving credential isolation, send-once settings, and redirect behavior. The completed static review identified no actionable regression.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 96a3427 · @DataDog review to ask questions}

dd-octo-sts · 2026-10-21T01:57:02Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-21T01:57:06Z

View all feedbacks in Devflow UI.

2026-10-21 01:57:06 UTC ℹ️ Start processing command /merge

2026-10-21 01:57:10 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-10-21 02:55:44 UTC ℹ️ MergeQueue: This merge request was merged

Keep the deletion of the agent-coupled Feature Flags writers modified by #12575, whose SDK identity headers are ported next, and of the feature-flagging-api module cleaned up by #12706. Restore the CODEOWNERS rules of other teams lost in an earlier merge.

Port #12575 to the SDK and the agent bridge: exposures and flag evaluations carry the DD-EVP-ORIGIN and DD-EVP-ORIGIN-VERSION headers, dd-trace-java and the library version, through the agent event proxy and direct intake alike. An agentless proxy endpoint is used only when the Agent forwards both headers. The smoke tests mock Agent advertises them like the Agent does.

Port #12575 to the SDK and the agent bridge: exposures and flag evaluations carry the DD-EVP-ORIGIN and DD-EVP-ORIGIN-VERSION headers, dd-trace-java and the library version, through the agent event proxy and direct intake alike. An agentless proxy endpoint is used only when the Agent forwards both headers. The smoke tests mock Agent advertises them like the Agent does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

feat(ffe): Restore the agent Feature Flags behaviors through the bridge Port the agent-side behaviors of the removed writers: send each event batch once with proxy discovery and recovery, and the Java SDK identity headers (#12477, #12575); flush the SDK pending events at agent shutdown, with or without tracing; declare the SDK settings the agent exposes (#12884, #12774). Move the Feature Flags shutdown tests from the agent tests to the OpenFeature smoke tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Guard OpenFeature agent bridge against incompatible SDK interfaces fix(ffe): Report rejected Remote Configuration flag documents as apply errors The SDK dropped malformed Remote Configuration documents and the agent swallowed subscriber failures, so the poller acknowledged documents the SDK never applied. The SDK now rejects malformed documents by throwing, and the agent rethrows the first subscriber failure once every subscriber is notified, so the poller reports an apply error as before the switch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Release the Remote Configuration poller when closing Feature Flags The poller reference-counts start and stop calls, but closing the Feature Flags Remote Configuration service no longer stopped it, keeping it polling once Feature Flags closed and leaking a reference on each restart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Flush Feature Flags at exit when the shutdown instrumentation is disabled The Feature Flags shutdown hook was only registered when tracing was disabled, but the tracing shutdown path also relies on the shutdown instrumentation, which can be disabled on its own. The hook is now always registered once the SDK subscribes to the shutdown; stopping is idempotent, so it does nothing after the tracing shutdown path already stopped Feature Flags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> test(ffe): Drop the agent bridge class loader test The test rewrote the SPI classes to check the runtime SPI guard. The japicmp bridge SPI check covers SPI compatibility once the SPI is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> chore(ffe): Mark the runtime bridge SPI check for removal after 1.69.0 Once dd-openfeature 1.69.0 is released, the japicmpBridgeV1 task checks the frozen bridge.v1 SPI, so the runtime check of the helpers becomes redundant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> chore(ffe): Track the checks to enable after the dd-openfeature 1.69.0 release Reference APMLP-1512 in the TODOs waiting for the first SDK release with the agent bridge, and prepare the muzzle pass directive of the bridge instrumentation, to enable once 1.69.0 is published. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> chore(ffe): Drop the TODO about Remote Configuration without the agent Remote Configuration without the Datadog Java agent is out of scope. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> fix(ffe): Keep event proxy available with blank API key (cherry picked from commit ef95143) fix(ffe): Defer RC until an SDK subscriber is validated (cherry picked from commit dc27b62) fix(ffe): Restore RC before SDK subscriber is validated (cherry picked from commit c8ab5f6) fix(ffe): Report a rejected Remote Configuration document received before the SDK The agent registers the Feature Flags product at startup, so the first document usually arrives before any SDK subscribes and is acknowledged without validation. When the first subscriber rejects it on replay, the service now registers the product again, which drops its cached documents, so the next poll delivers the document again and reports the rejection as an apply error, without depending on the Remote Configuration core. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> test(ffe): Check a malformed first Remote Configuration document is reported The smoke application can now register its provider on request instead of at startup, so the test delivers a malformed flag configuration before the SDK subscribes, then checks the agent reports it as not applied once the SDK rejects it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> docs(ffe): Qualify when the agent shutdown notifies the SDK The agent shutdown notifies the SDK before the JVM shutdown hooks only when the tracing shutdown instrumentation runs, otherwise from a JVM shutdown hook. Document what always holds: the notification comes before the agent releases its Feature Flags resources. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> test(ffe): Check the shutdown flush with a single smoke test The Feature Flagging shutdown hook is now always registered once the SDK subscribes, whether tracing is enabled or not, so the tracing disabled variant tested the same behavior. Keep one test, with the default setup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: sarahchen6 <sarah.chen@datadoghq.com> Co-authored-by: bruce.bujon <bruce.bujon@datadoghq.com>

leoromanovsky added 4 commits September 12, 2026 03:50

Harden agentless Feature Flags EVP delivery

612ef52

Add capability-gated local discovery, safe sticky direct fallback, shared route state, send-once semantics, and bounded lifecycle handling for Java Feature Flags telemetry. Environment: Datadog workspace

test(openfeature): cover EVP fallback edge states

e20a9d7

leoromanovsky added type: feature Enhancements and improvements tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Sep 19, 2026

leoromanovsky mentioned this pull request Sep 19, 2026

Harden Agentless Feature Flags EVP delivery #12477

Merged

leoromanovsky mentioned this pull request Sep 23, 2026

test(java): enable agentless EVP egress DataDog/system-tests#7702

Draft

leoromanovsky added 2 commits October 7, 2026 01:55

Merge master into Java EVP fallback hardening

909f9ab

Environment: Datadog workspace

leoromanovsky added this pull request to stack #12771 October 7, 2026 15:24

leoromanovsky marked this pull request as ready for review October 7, 2026 15:25

leoromanovsky requested review from a team as code owners October 7, 2026 15:25

leoromanovsky requested review from greghuels, mcculls and vjfridge and removed request for a team October 7, 2026 15:25

datadog-datadog-prod-us1 Bot reviewed Oct 7, 2026

View reviewed changes