feat(wall): support signal suppression for unfiltered threads by kaahos · Pull Request #663 · DataDog/java-profiler

kaahos · 2026-07-16T13:30:31Z

What does this PR do?:

Extends wall-clock signal suppression (wallprecheck=true) to recordings that run with an explicit empty filter=, i.e. context filtering disabled and every thread sampled. (Omitting filter still defaults to "0", which enables context filtering; the meaning of filter= is unchanged.)

When wall profiling runs with both filter= and wallprecheck=true, the profiler now:

Tracks thread identity and lifecycle (native TID, lifecycle generation, recording epoch) independently from context-window membership.
Binds every thread's registry slot lazily, on its first filterThreadAdd0 / parkEnter0 / blockEnter0 call. ThreadStart does not pre-register threads in this mode (nor does start() register its caller), so idle threads that never use those hooks don't consume one of the bounded registry slots. The wall timer only looks slots up; it never allocates them.
Retires identities on thread teardown through JVMTI ThreadEnd and, because ThreadEnd notifications are disabled between recordings, also through the ProfiledThread TLS destructor, so a reused TID can't inherit a stale slot.
Lets both the ASGCT and JVMTI wall-clock engines (HotSpot) use this metadata on the timer thread and in the signal handler.
Suppresses repeated samples only for profiler-owned blocking runs (park/block hooks) that started and stayed outside the context window. Threads inside a context window keep their normal per-signal MethodSample stream.
Keeps blocked threads that aren't in a profiler-owned block on ordinary per-signal sampling in this mode (no 1-in-N unowned decimation).
Validates thread identity, slot lifecycle, recording epoch and context-window transitions before suppressing anything.

Motivation:

With filter=, wall-clock profiling already samples every thread, but threads outside a context window had no lifecycle metadata, so the profiler could not recognize repeated samples of the same profiler-owned blocking run. This closes that gap and provides the thread-lifecycle machinery needed by the follow-up TaskBlock work for non-context threads.

Implementation notes:

Registry (ThreadFilter): slots are keyed by native TID through a lock-free open-addressing index (8192 entries, allocated on first activation). Writers are serialized by a mutex that never runs in a signal handler; lookups stay lock-free. Capacity is still 2048 threads.
Generations and epochs: lifecycle generations stop reused slots/TIDs from inheriting stale state. A per-recording epoch invalidates slots across recordings. Context-window epochs (packed with the in-window bit) block suppression if a thread enters or leaves a context during a blocking run.
WallClockBlockTracker: the owned-block suppression state moved out of ThreadFilter::Slot into a separate per-slot array, kept parallel to the registry.
Timer selection in unfiltered mode: candidates come from OS::listThreads(). The timer walks a randomized prefix (bounded to 4× the reservoir size) and backfills past suppressed threads, so suppression doesn't use up the per-tick signal budget and no thread is favored by list order.
Activation: registry tracking turns on only for wall recordings with filter= + wallprecheck=true on an engine that supports it (supportsUnfilteredThreadRegistryTracking(); the J9 JVMTI engine does not). It is turned off again on stop, when wall startup fails while other engines succeed, on any other start failure, and when a later recording uses a different configuration.
New counters: thread_registry_capacity_exhausted, thread_registry_index_failures, thread_registry_context_reset_race_detected, thread_registry_hook_reregistration, wc_precheck_registry_lookups, wc_precheck_candidates_rejected, wc_precheck_lookup_budget_exhausted.

Behavior changes in context-filtered mode (filter set to a non-empty value, including the default "0"):

With wallprecheck=true, suppressed threads are now removed before reservoir sampling, so they no longer use up reservoir slots; other in-context threads may be signalled more often per tick.
WallClockEpoch.numSuppressedSampledRun now counts every suppressed thread in the pool on each tick, not only the ones the reservoir selected. samplePoolSize still counts all candidates.
Entering a context window no longer resets unflushed blocked-sample weight from an earlier window; that weight is emitted on the next running sample instead of being dropped.
Thread registration and teardown now go through the TID index: JVMTI ThreadStart/ThreadEnd take _registry_lock (a mutex, never taken in a signal handler) where registration was previously lock-free. Lookups and the add()/remove() hot path stay lock-free.
add() now validates that the slot still belongs to the calling thread's TID, and clears any stuck active block run (e.g. an exit missed on an exception path) on context entry.

Memory: WallClockBlockTracker is a fixed 2048 × 64 B array (~128 KB) allocated with the profiler singleton and accounted under NM_WALLCLOCK. The TID index (~48 KB: 8192 entries + probe counts) is allocated by the first recording that activates the registry, i.e. any context-filtered recording (including the default filter="0") or an unfiltered wall-precheck recording, and is accounted under NM_THREAD_FILTER.

Also included:

BaseWallClock::stop() no longer calls pthread_kill/pthread_join on a thread that was never created (crashed on musl after a failed wall start).
Minor refactor of copyJvmtiFrames: reads method/location into locals before writing either output field (no behavior change — the previous loop already did this; the two buffers are views of the same union, so this is just making the existing read-before-write ordering explicit).
Profiler::stop() now gates engine teardown on the engines that actually activated (_activated_mask) instead of the requested _event_mask, so a mixed-success start() no longer calls stop() on an engine whose start() failed.

How to test the change?:

Unit tests: ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug_threadFilter_ut :ddprof-lib:gtestDebug_wallClockBlockTracker_ut :ddprof-lib:gtestDebug_wallClockCandidateSelector_ut :ddprof-lib:gtestDebug_park_state_ut :ddprof-lib:gtestDebug_wallprecheck_args_ut
Stress (Linux): :ddprof-lib:gtestDebug_stress_wallClockBlockTracker_ut
Chaos coverage with prechecks on and off: ./.claude/commands/build-and-summarize :ddprof-stresstest:testOwnedBlockPrecheck
Integration: ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests="*UnfilteredWallPrecheck*" and -Ptests=ConcurrentOwnedBlockChurnTest
Full suite: ./.claude/commands/build-and-summarize testDebug
Fuzzing: fuzz_threadFilter (see ddprof-lib/src/test/fuzz/README.md)
Chaos: park-block-churn antagonist (utils/run-chaos-harness.sh)
Overhead: JMH WallClockPrecheckOverheadBenchmark

For Datadog employees:

If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a security review.
This PR doesn't touch any of that.
JIRA: [PROF-XXXX]

dd-octo-sts · 2026-07-16T13:51:39Z

CI Test Results

Run: #39601202496 | Commit: 247ad3e | Duration: 18m 55s (longest job)

✅ All 76 test jobs passed

Status Overview

JDK	glibc-aarch64/asan	glibc-aarch64/debug	glibc-aarch64/tsan	glibc-amd64/asan	glibc-amd64/debug	glibc-amd64/tsan	musl-aarch64/debug	musl-amd64/debug
8	-	-	-	✅	✅	✅	-	-
8-ibm	-	-	-	✅	✅	✅	-	-
8-j9	✅	✅	✅	✅	✅	✅	-	-
8-librca	-	-	-	-	-	-	✅	✅
8-orcl	-	-	-	✅	✅	✅	-	-
11	-	-	-	✅	✅	✅	-	-
11-j9	✅	✅	✅	✅	✅	✅	-	-
11-librca	-	-	-	-	-	-	✅	✅
17	✅	✅	✅	✅	✅	✅	-	-
17-graal	✅	✅	✅	✅	✅	✅	-	-
17-j9	✅	✅	✅	✅	✅	✅	-	-
17-librca	-	-	-	-	-	-	✅	✅
21	✅	✅	✅	✅	✅	✅	-	-
21-graal	✅	✅	✅	✅	✅	✅	-	-
21-librca	-	-	-	-	-	-	✅	✅
25	✅	✅	✅	✅	✅	✅	-	-
25-graal	✅	✅	✅	✅	✅	✅	-	-
25-librca	-	-	-	-	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 76 | Passed: 76 | Failed: 0

Updated: 2026-10-21 13:00:18 UTC

dd-octo-sts · 2026-07-19T19:14:40Z

Benchmark Results (commit `fb1c44d`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125547869 Commit: fb1c44de77197780eae60267ff98bd38a907a558

⚠️ Significant outliers

🔴 fj-kmeans (JDK 21): runtime +5.9% (2679→2836 ms)
🔴 future-genetic (JDK 21): runtime +3.8% (2083→2162 ms)

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10210 ms (21 iters)	✅ 10256 ms (21 iters)	≈ +0.5% (±10.9%)	— / —
akka-uct	25	✅ 8953 ms (24 iters)	✅ 8890 ms (24 iters)	≈ -0.7% (±10.2%)	— / —
finagle-chirper	21	✅ 6000 ms (33 iters)	✅ 5960 ms (33 iters)	≈ -0.7% (±25.4%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5431 ms (36 iters)	✅ 5443 ms (36 iters)	≈ +0.2% (±24.6%)	⚠️ W:4 / ⚠️ W:3
fj-kmeans	21	✅ 2679 ms (70 iters)	✅ 2836 ms (66 iters)	🔴 +5.9%	— / —
fj-kmeans	25	✅ 2847 ms (66 iters)	✅ 2828 ms (66 iters)	≈ -0.7% (±2.6%)	— / —
future-genetic	21	✅ 2083 ms (89 iters)	✅ 2162 ms (86 iters)	🔴 +3.8%	— / —
future-genetic	25	✅ 2076 ms (90 iters)	✅ 2122 ms (88 iters)	≈ +2.2% (±2.5%)	— / —
naive-bayes	21	✅ 1252 ms (136 iters)	✅ 1248 ms (137 iters)	≈ -0.3% (±32.7%)	— / —
naive-bayes	25	✅ 1008 ms (170 iters)	✅ 1023 ms (167 iters)	≈ +1.5% (±31.7%)	— / —
reactors	21	✅ 16312 ms (15 iters)	✅ 16357 ms (15 iters)	≈ +0.3% (±8.8%)	— / —
reactors	25	✅ 18416 ms (15 iters)	✅ 18780 ms (15 iters)	≈ +2% (±3.6%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	2 / ✅	1891 / 2004	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	1 / 1	2305 / 2283	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	4 / 2	8183 / 8674	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	✅ / ✅	8087 / 8641	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	2 / 2	1270 / 1278	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	3 / 3	1282 / 1275	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	1 / 3	2966 / 3007	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	2 / ✅	2992 / 2972	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	3 / 6	3491 / 3501	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	1 / 5	3488 / 3493	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	2 / ✅	1711 / 1786	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	✅ / ✅	1819 / 1920	✅ / ✅	✅ / ✅

dd-octo-sts · 2026-07-19T21:44:20Z

Benchmark Results (commit `1118f62`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125554643 Commit: 1118f62831a4384bdf0283ef01a626c5bdc48af5

⚠️ Significant outliers

🟢 fj-kmeans (JDK 21): runtime -3.9% (2808→2698 ms)
🟢 future-genetic (JDK 25): runtime -4.2% (2088→2001 ms)

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10294 ms (21 iters)	✅ 10251 ms (21 iters)	≈ -0.4% (±11.4%)	— / —
akka-uct	25	✅ 8933 ms (24 iters)	✅ 8835 ms (24 iters)	≈ -1.1% (±10%)	— / —
finagle-chirper	21	✅ 6016 ms (33 iters)	✅ 6034 ms (33 iters)	≈ +0.3% (±25.3%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5462 ms (36 iters)	✅ 5452 ms (36 iters)	≈ -0.2% (±24.6%)	⚠️ W:3 / ⚠️ W:3
fj-kmeans	21	✅ 2808 ms (66 iters)	✅ 2698 ms (70 iters)	🟢 -3.9%	— / —
fj-kmeans	25	✅ 2812 ms (66 iters)	✅ 2813 ms (66 iters)	≈ +0% (±2.6%)	— / —
future-genetic	21	✅ 2067 ms (90 iters)	✅ 2083 ms (90 iters)	≈ +0.8% (±2.7%)	— / —
future-genetic	25	✅ 2088 ms (89 iters)	✅ 2001 ms (93 iters)	🟢 -4.2%	— / —
naive-bayes	21	✅ 1239 ms (138 iters)	✅ 1231 ms (138 iters)	≈ -0.6% (±32.7%)	— / —
naive-bayes	25	✅ 1019 ms (168 iters)	✅ 1011 ms (169 iters)	≈ -0.8% (±32.1%)	— / —
reactors	21	✅ 16149 ms (15 iters)	✅ 16215 ms (15 iters)	≈ +0.4% (±8.4%)	— / —
reactors	25	✅ 18598 ms (15 iters)	✅ 18345 ms (15 iters)	≈ -1.4% (±5.1%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	2 / 2	1978 / 1973	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	2 / ✅	2287 / 2449	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	2 / 2	8978 / 8724	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	✅ / ✅	8633 / 8297	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	2 / 1	1257 / 1281	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	1 / 4	1280 / 1268	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	✅ / 3	2909 / 2923	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	2 / 5	2847 / 2887	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	7 / 4	3492 / 3493	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	4 / 2	3463 / 3473	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	✅ / 1	1761 / 1721	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	✅ / ✅	1916 / 1880	✅ / ✅	✅ / ✅

dd-octo-sts · 2026-07-20T08:43:08Z

Benchmark Results (commit `40dbe8e`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125596529 Commit: 40dbe8e833888192b6026558590bce1cf0ad1c9f

⚠️ Significant outliers

🟢 fj-kmeans (JDK 21): runtime -2.7% (2823→2748 ms)
🔴 future-genetic (JDK 21): runtime +4.1% (2041→2125 ms)
🔴 future-genetic (JDK 25): runtime +5.8% (1994→2110 ms)

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10340 ms (21 iters)	✅ 10312 ms (21 iters)	≈ -0.3% (±11.6%)	— / —
akka-uct	25	✅ 8871 ms (24 iters)	✅ 8849 ms (24 iters)	≈ -0.2% (±10.1%)	— / —
finagle-chirper	21	✅ 6017 ms (33 iters)	✅ 6015 ms (33 iters)	≈ -0% (±25%)	⚠️ W:3 / ⚠️ W:4
finagle-chirper	25	✅ 5470 ms (36 iters)	✅ 5509 ms (36 iters)	≈ +0.7% (±24.2%)	⚠️ W:4 / ⚠️ W:3
fj-kmeans	21	✅ 2823 ms (66 iters)	✅ 2748 ms (68 iters)	🟢 -2.7%	— / —
fj-kmeans	25	✅ 2856 ms (66 iters)	✅ 2847 ms (66 iters)	≈ -0.3% (±2.6%)	— / —
future-genetic	21	✅ 2041 ms (90 iters)	✅ 2125 ms (88 iters)	🔴 +4.1%	— / —
future-genetic	25	✅ 1994 ms (93 iters)	✅ 2110 ms (88 iters)	🔴 +5.8%	— / —
naive-bayes	21	✅ 1247 ms (137 iters)	✅ 1259 ms (135 iters)	≈ +1% (±33.5%)	— / —
naive-bayes	25	✅ 1027 ms (167 iters)	✅ 1018 ms (168 iters)	≈ -0.9% (±31.5%)	— / —
reactors	21	✅ 16583 ms (15 iters)	✅ 16798 ms (15 iters)	≈ +1.3% (±7.5%)	— / —
reactors	25	✅ 18582 ms (15 iters)	✅ 18610 ms (15 iters)	≈ +0.2% (±5.1%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	✅ / 6	1978 / 1998	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	✅ / 1	2344 / 2271	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	2 / 2	8863 / 8312	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	1 / ✅	8368 / 8738	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	7 / 1	1284 / 1242	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	1 / 4	1302 / 1289	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	1 / 1	2979 / 2980	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	✅ / ✅	2847 / 2862	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	4 / 5	3520 / 3534	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	3 / 3	3523 / 3509	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	2 / 2	1682 / 1809	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	2 / ✅	1806 / 1871	✅ / ✅	✅ / ✅

Copilot

Pull request overview

Extends wall-clock “owned-block” signal suppression to unfiltered (explicit filter=) wall recordings when wallprecheck=true, by introducing a thread registry keyed by native TID and integrating it into both ASGCT and JVMTI wall-clock sampling paths.

Changes:

Added unfiltered wall thread registry support (native TID indexing, lifecycle generations, recording epochs) and wired it into profiler lifecycle + wall-clock precheck logic.
Added a bounded candidate-selection/backfill mechanism for precheck to avoid reverting to O(N) registry work per wall tick under heavy suppression.
Added Java integration tests, C++ unit tests, and a JMH benchmark to validate behavior and measure overhead.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java	New integration coverage for suppression behavior in explicit-empty-filter wall recordings.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java	Verifies registry activation does not leak across recording restarts and handles stopped gaps safely.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java	Ensures the unfiltered path works when wall sampling delegates stacks via JVMTI.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java	Confirms J9 wall engine capability gating prevents activation of unfiltered tracking.
ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java	Adds `beforeProfilerStart()` hook to allow test setup before profiler thread callbacks are enabled.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java	Exposes owned-block hooks to benchmarks without widening core API surface.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java	Adds throughput benchmark comparing `wallprecheck=false/true` as owned-block population grows.
ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp	Adds capability tests and verifies explicit-empty vs omitted `filter` parsing behavior.
ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp	Unit-tests the new bounded candidate selection logic.
ddprof-lib/src/test/cpp/threadFilter_ut.cpp	Expands tests for registry/epoch/lifecycle behavior and adjusts TID usage in recovery test.
ddprof-lib/src/main/cpp/wallClockCandidateSelector.h	Adds reusable “randomized prefix without replacement” candidate selection with stats.
ddprof-lib/src/main/cpp/wallClock.h	Integrates candidate selector and adds bounded-visit backfill path to the wall timer loop.
ddprof-lib/src/main/cpp/wallClock.cpp	Updates precheck suppression to validate registry identity + context transitions; supports lazy registry lookups.
ddprof-lib/src/main/cpp/threadFilter.h	Introduces registry state (TID index, epochs, lifecycle generation) and APIs for unfiltered tracking.
ddprof-lib/src/main/cpp/threadFilter.cpp	Implements lock-free lookups + mutex-serialized writers for TID indexing, epoch refresh, and retirement.
ddprof-lib/src/main/cpp/profiler.h	Declares bootstrap of existing Java threads for registry population.
ddprof-lib/src/main/cpp/profiler.cpp	Boots existing Java threads into registry when unfiltered tracking is active; deactivates registry on stop/failure.
ddprof-lib/src/main/cpp/jvmThread.h	Adds capability check for cross-thread native TID lookup support.
ddprof-lib/src/main/cpp/jvmThread.cpp	Implements native TID lookup support predicate.
ddprof-lib/src/main/cpp/javaApi.cpp	Ensures TLS slot binding is validated/refreshed before using owned-block and filter APIs.
ddprof-lib/src/main/cpp/engine.h	Adds wall-engine capability hook for unfiltered precheck support.
ddprof-lib/src/main/cpp/counters.h	Adds counters for registry bootstrap and bounded-lookup/backfill diagnostics.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dd-octo-sts · 2026-07-21T07:47:51Z

Benchmark Results (commit `d82291f`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125875202 Commit: d82291ff62852b63091350a05ebf26283fa39029

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10141 ms (21 iters)	✅ 10278 ms (21 iters)	≈ +1.4% (±11.9%)	— / —
akka-uct	25	✅ 8800 ms (24 iters)	✅ 8939 ms (24 iters)	≈ +1.6% (±9.7%)	— / —
finagle-chirper	21	✅ 6003 ms (33 iters)	✅ 6060 ms (33 iters)	≈ +0.9% (±25.4%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5463 ms (36 iters)	✅ 5482 ms (36 iters)	≈ +0.3% (±24.1%)	⚠️ W:3 / ⚠️ W:3
fj-kmeans	21	✅ 2780 ms (68 iters)	✅ 2771 ms (68 iters)	≈ -0.3% (±2.7%)	— / —
fj-kmeans	25	✅ 2752 ms (68 iters)	✅ 2824 ms (66 iters)	≈ +2.6% (±2.7%)	— / —
future-genetic	21	✅ 2082 ms (90 iters)	✅ 2035 ms (91 iters)	≈ -2.3% (±2.6%)	— / —
future-genetic	25	✅ 2101 ms (88 iters)	✅ 2055 ms (90 iters)	≈ -2.2% (±2.6%)	— / —
naive-bayes	21	✅ 1299 ms (132 iters)	✅ 1232 ms (138 iters)	≈ -5.2% (±31.7%)	— / —
naive-bayes	25	✅ 1011 ms (169 iters)	✅ 1013 ms (169 iters)	≈ +0.2% (±31.4%)	— / —
reactors	21	✅ 16511 ms (15 iters)	✅ 16445 ms (15 iters)	≈ -0.4% (±9.8%)	— / —
reactors	25	✅ 18516 ms (15 iters)	✅ 18146 ms (15 iters)	≈ -2% (±6.2%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	2 / 2	1925 / 1914	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	2 / 3	2117 / 2331	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	4 / 3	8401 / 8842	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	1 / 1	8557 / 8073	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	2 / 3	1243 / 1272	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	✅ / 1	1265 / 1283	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	3 / 1	3054 / 3036	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	✅ / 1	2890 / 2867	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	4 / 3	3522 / 3464	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	4 / 5	3452 / 3463	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	✅ / 2	1679 / 1552	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	✅ / ✅	1933 / 1805	✅ / ✅	✅ / ✅

dd-octo-sts · 2026-07-21T08:27:57Z

Benchmark Results (commit `ec76f58`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125883187 Commit: ec76f5829d243f516dee70833f3dfbe7a2b9eaa3

⚠️ Significant outliers

🟢 fj-kmeans (JDK 21): runtime -4.1% (2839→2724 ms)
🔴 future-genetic (JDK 21): runtime +3% (2066→2127 ms)
🔴 future-genetic (JDK 25): runtime +5.8% (2006→2122 ms)

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10350 ms (21 iters)	✅ 10305 ms (21 iters)	≈ -0.4% (±11.3%)	— / —
akka-uct	25	✅ 8864 ms (24 iters)	✅ 8976 ms (24 iters)	≈ +1.3% (±10.4%)	— / —
finagle-chirper	25	✅ 5472 ms (36 iters)	✅ 5418 ms (36 iters)	≈ -1% (±24.3%)	⚠️ W:3 / ⚠️ W:3
fj-kmeans	21	✅ 2839 ms (66 iters)	✅ 2724 ms (68 iters)	🟢 -4.1%	— / —
fj-kmeans	25	✅ 2851 ms (66 iters)	✅ 2836 ms (66 iters)	≈ -0.5% (±2.6%)	— / —
future-genetic	21	✅ 2066 ms (89 iters)	✅ 2127 ms (87 iters)	🔴 +3%	— / —
future-genetic	25	✅ 2006 ms (92 iters)	✅ 2122 ms (87 iters)	🔴 +5.8%	— / —
naive-bayes	21	✅ 1277 ms (134 iters)	✅ 1289 ms (133 iters)	≈ +0.9% (±32.4%)	— / —
naive-bayes	25	✅ 1021 ms (167 iters)	✅ 1023 ms (168 iters)	≈ +0.2% (±31.6%)	— / —
reactors	21	✅ 15903 ms (15 iters)	✅ 16452 ms (15 iters)	≈ +3.5% (±8.8%)	— / —
reactors	25	✅ 18032 ms (15 iters)	✅ 18628 ms (15 iters)	≈ +3.3% (±4.7%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	1 / 3	2089 / 2428	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	7 / 2	8520 / 8487	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	4 / 2	8382 / 8319	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	4 / 4	1278 / 1277	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	4 / ✅	1290 / 1289	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	3 / 2	2953 / 3032	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	2 / 2	2846 / 2856	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	3 / 3	3534 / 3561	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	1 / 3	3480 / 3510	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	1 / ✅	1614 / 1718	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	✅ / 3	1962 / 1876	✅ / ✅	✅ / ✅

…-all-threads

kaahos · 2026-10-01T16:10:04Z

@jbachorik I've addressed the latest review comments and added some fuzzing and chaos tests. Can you please re review?

rkennke

A few comments inline. The main one is the dropped active-block reset in filterThreadAdd0. It's latent until the TaskBlock hooks are wired up, but cheap to fix now. The rest are small.

rkennke · 2026-10-02T17:39:11Z

About the failing glibc-aarch64/asan / 25 cell: it isn't caused by this PR. Both attempts' asan-logs-* artifact holds the same UBSan report:

stackWalker.inline.h:71:41: runtime error: applying non-zero offset to non-null pointer 0x000000000001 produced null pointer
  #0 attributionPC(void const*, bool)
  #1 Profiler::resolveNativeFrameForWalkVM   profiler.cpp:403
  #2 HotspotSupport::walkVM                  hotspotSupport.cpp:800

It fires from both the CPU and wall-clock signal handlers. walkVM can read a garbage return address (here 0x1), and attributionPC()'s (const char*)pc - 1 then becomes pointer arithmetic that produces null, which is UB. The sanitized test config logs to /tmp/asan.log with halt_on_error=0, so no test fails; the test JVM just exits with code 1 at shutdown.

main has the same problem (the Nightly Sanitized Run reports the pc == nullptr variant). The pc != nullptr guard merged in here from fix/nightlies_sanitized covers that variant but not 0x1.

Fixed separately in #835, which does the subtraction with uintptr_t arithmetic and adds a regression test. Once #835 lands, could you drop the attributionPC guard from this PR and rebase?

jbachorik · 2026-10-05T12:52:26Z

+      std::mt19937 candidate_generator;
+      if (lazyBackfill) {
+        candidate_generator.seed(
+            xorshift::seed((u64)(uintptr_t)&candidate_generator, TSC::ticks()));
+      }


There was a PR cleaning up various RNG devices (#794) - perhaps, this needs to be adjusted to how we are using the random generators now.

jbachorik · 2026-10-05T12:52:26Z

+    // A plain load + release store per call should stay well under a locked
+    // RMW's cost; this is a loose ceiling to catch a regression back to CAS
+    // or worse, not a tight performance contract.
+    EXPECT_LT(ns_per_op, 50.0);


This could become flaky due to different CI machines etc. I wonder if it should be actually failing test because of that. Maybe have a base workload and compare base vs. /w filter.add/remove ?

fixed in b77821f

jbachorik · 2026-10-05T12:52:29Z

A few minor comments, but I think we are getting close to the final state. Thanks for the patience!

kaahos · 2026-10-13T16:03:09Z

thanks for the reviews @rkennke @jbachorik! I've addressed the comments, it would be great if you could re-review at some point.

kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 3 times, most recently from 3902a14 to f302dd5 Compare July 19, 2026 12:00

kaahos changed the title ~~feat(wall): add configurable wall thread scope~~ feat(wall): support signal suppression for unfiltered threads Jul 19, 2026

kaahos force-pushed the paul.fournillon/wallclock-all-threads branch 2 times, most recently from 0ad425c to fb1c44d Compare July 19, 2026 18:43

feat(wall): track all threads for unfiltered wall-clock precheck

1118f62

kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from fb1c44d to 1118f62 Compare July 19, 2026 21:16

Merge branch 'main' into paul.fournillon/wallclock-all-threads

40dbe8e

Merge branch 'main' into paul.fournillon/wallclock-all-threads

d82291f

Copilot AI review requested due to automatic review settings July 21, 2026 07:17

Copilot started reviewing on behalf of kaahos July 21, 2026 07:18

View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread ddprof-lib/src/main/cpp/wallClockCandidateSelector.h

Comment thread ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp

Comment thread ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java

fix: apply review suggestions

ec76f58

Merge branch 'main' into paul.fournillon/wallclock-all-threads

52def55

DataDog deleted a comment from dd-octo-sts Bot Jul 21, 2026

kaahos added 4 commits September 30, 2026 18:32

fix: add chaos and fuzzing tests

b158ab9

Merge branch 'fix/nightlies_sanitized' into paul.fournillon/wallclock…

9a18895

…-all-threads

fix: address nits

d6f6e9c

fix: fix wrong comments

73b532d

Merge branch 'main' into paul.fournillon/wallclock-all-threads

6ae8dda

rkennke reviewed Oct 2, 2026

View reviewed changes

Comment thread ddprof-lib/src/main/cpp/javaApi.cpp

Comment thread ddprof-lib/src/main/cpp/wallClock.h

Comment thread ddprof-lib/src/main/cpp/frames.h

Comment thread ddprof-lib/src/main/cpp/threadFilter.cpp Outdated

Comment thread ddprof-lib/src/main/cpp/profiler.h

rkennke mentioned this pull request Oct 2, 2026

Compute the attribution pc with integer arithmetic #835

Merged

1 of 3 tasks

jbachorik reviewed Oct 5, 2026

View reviewed changes

fix: address review comments

fa0e33d

kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from dbacfe1 to fa0e33d Compare October 8, 2026 08:40

kaahos mentioned this pull request Oct 8, 2026

Fix duplicate and biased thread picks in the wall-clock reservoir sampler #844

Merged

1 of 3 tasks

test: move flaky self-calibrating ThreadFilter perf tests to a benchmark

b77821f

kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from 3b6ea1e to b77821f Compare October 9, 2026 07:29

kaahos added 3 commits October 9, 2026 11:12

Merge branch 'main' into paul.fournillon/wallclock-all-threads

aef6d90

Merge branch 'main' into paul.fournillon/wallclock-all-threads

b13fd76

Merge branch 'main' into paul.fournillon/wallclock-all-threads

c433cb4

kaahos requested review from jbachorik and rkennke October 13, 2026 12:47

kaahos added 3 commits October 20, 2026 18:42

fix: retain thread identities across recording restarts

5018858

fix: preserve wall-clock memory accounting across timer restarts

7124b6c

perf: register unfiltered wall-clock threads lazily

70230b4

kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from ef911c6 to 70230b4 Compare October 20, 2026 21:14

kaahos added 3 commits October 21, 2026 13:39

fix: address TID bug

926fd3a

fix: enforce weak tests and chaos tests

7928e27

Merge branch 'main' into paul.fournillon/wallclock-all-threads

b1cda93

kaahos force-pushed the paul.fournillon/wallclock-all-threads branch from 8dcb2b6 to b1cda93 Compare October 21, 2026 12:38

Conversation

kaahos commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI Test Results

Status Overview

Uh oh!

dd-octo-sts Bot commented Jul 19, 2026

Benchmark Results (commit fb1c44d)

⚠️ Significant outliers

Uh oh!

dd-octo-sts Bot commented Jul 19, 2026

Benchmark Results (commit 1118f62)

⚠️ Significant outliers

Uh oh!

dd-octo-sts Bot commented Jul 20, 2026

Benchmark Results (commit 40dbe8e)

⚠️ Significant outliers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026

Benchmark Results (commit d82291f)

✅ Within expected boundaries

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026

Benchmark Results (commit ec76f58)

⚠️ Significant outliers

Uh oh!

kaahos commented Oct 1, 2026

Uh oh!

rkennke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rkennke commented Oct 2, 2026

Uh oh!

Uh oh!

jbachorik Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

kaahos Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

jbachorik Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

kaahos Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jbachorik commented Oct 5, 2026

Uh oh!

kaahos commented Oct 13, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kaahos commented Jul 16, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 16, 2026 •

edited

Loading

Benchmark Results (commit `fb1c44d`)

Benchmark Results (commit `1118f62`)

Benchmark Results (commit `40dbe8e`)

Benchmark Results (commit `d82291f`)

Benchmark Results (commit `ec76f58`)