Repository navigation
Conversation
| // G1/Serial/Parallel relocate objects at safepoints. A sampled Java thread | ||
| // cannot reach a safepoint while its signal handler is reading these oops. | ||
| // Concurrently relocating collectors require load barriers not provided here. |
There was a problem hiding this comment.
Can the profiling signal land on a thread with frozen stack and already in safepoint?
Or is this structurally impossible?
There was a problem hiding this comment.
yes it can, it is not structurally excluded. There is a protection in FrozenStackWalker::append: it rejects any thread state other than _thread_in_Java before reading heap objects. HotSpot switches to _thread_blocked before waiting at the safepoint, so those samples skip frozen-frame recovery and retain truncated=true. For accepted samples, the PC must also belong to interpreted or compiled Java code, excluding freeze/thaw runtime execution.
CI Test ResultsRun: #39815407680 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled
|
| Test | Cells | Message |
|---|---|---|
FrozenVirtualThreadStackTest.cstack_vm |
glibc-21-graal-debug-aarch64 |
org.opentest4j.AssertionFailedError: Samples omitted the JVM-confirmed outer caller; recovered=1318, failed=9 ==> expect |
These fail the build. Passing on a second run makes a test flaky, not passing. Fix it, or quarantine it against a ticket so the debt is tracked rather than forgotten.
Consider quarantining these — click for ready-made entries
A quarantined test still runs and still reports; its failures just stop
turning CI red. To quarantine one:
- Open a PROF ticket for the test, linking the failing job.
- Append the line below to
ddprof-test/quarantine.txt, replacing
PROF-XXXXXwith the ticket number. - Check the
cellsandreasoncolumns — the proposal only knows what
failed in this run, and a narrowercellsglob keeps the same test
gating everywhere it has not misbehaved.
CI fails once review_by passes, so an entry expires instead of piling up.
# test | ticket | added | review_by | cells | reason
# WIDENED: observed `com.datadoghq.profiler.wallclock.FrozenVirtualThreadStackTest.cstack=vm`, which covers() cannot match exactly; this entry mutes the whole class
com.datadoghq.profiler.wallclock.FrozenVirtualThreadStackTest.* | PROF-XXXXX | 2026-10-22 | 2027-01-20 | glibc-21-graal-debug-aarch64 | org.opentest4j.AssertionFailedError: Samples omitted the JVM-confirmed outer caller; recovered=1318, failed=9 ==> expected: <0> but was: <1> (seen in: glibc-21-graal-debug-aarch64)
❌ Failing tests
| Test | Cells | Message |
|---|---|---|
VirtualThreadWallClockTest.cstack_vmx |
glibc-21-debug-aarch64, glibc-25-debug-aarch64, glibc-21-graal-debug-amd64 |
org.opentest4j.AssertionFailedError: No sample showed carrier-thread frames (ForkJoinWorkerThread) — continuation unwind |
Retried 4 of 32 cells.
Failed Jobs
- glibc-aarch64/debug / 21 — view logs
- glibc-aarch64/debug / 21-graal — view logs
- glibc-aarch64/debug / 25 — view logs
- glibc-amd64/debug / 21-graal — view logs
Summary: Total: 32 | Passed: 28 | Failed: 4
Updated: 2026-10-22 17:05:57 UTC
…nto paul.fournillon/recover-frozen-virtual-thread-callers
Reliability & Chaos Results✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/146813861 |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nested continuation transitions can make frozen-stack recovery append suspended child callers instead of parent callers and incorrectly report the trace as complete.
🤖 Bits Code Review · Commit 746acbe
| // carrier Thread.cont, including when the current Java thread is virtual. | ||
| uintptr_t continuation = readOop(carrier, layout.carrier_cont); | ||
| for (unsigned int i = 0; i < continuation_depth && continuation != 0; i++) { | ||
| continuation = readOop(continuation, layout.cont_parent); |
There was a problem hiding this comment.
When a partially thawed parent resumes a nested continuation, Continuation.run publishes the child in Thread.cont before enterSpecial and restores the parent only after it returns. Samples in those Java intervals pass the recovery guards, but the first native return barrier belongs to the parent. Reading Thread.cont therefore selects the child's chunks, potentially appending suspended child frames, omitting parent callers, and clearing truncation. Match recovery to the crossed native continuation boundary, or reject recovery during these transitions. Correctly identifying that boundary requires broader changes, so no localized suggestion is provided.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
This branch has not been deployed
2 participants
What does this PR do?:
Recover older Java callers from HotSpot heap stack chunks when a mounted virtual thread reaches a continuation return barrier. Append interpreted and compiled frames, including inlined callers and parent chunks, to the existing trace buffer before stopping at the continuation root (
cstack=vm) or walking the carrier (cstack=vmx). For deoptimized compiled frames, recover the saved original return PC before decoding inline scopes.Extend regression coverage for both walking modes, CPU and wall sampling, parent chunks, mounted-frame deoptimization, depth limits, native calls, unsupported recovery, and platform-thread/fully thawed controls.
Motivation:
HotSpot can thaw only part of a virtual thread's stack after remounting it. The remaining Java callers stay in heap chunks and are absent from a walk of the carrier's native stack. #877 correctly marks that omission as truncated; this change restores those callers where recovery is supported.
Additional Notes:
paul.fournillon/fix-frozen-virtual-thread-truncation). Retarget tomainafter that dependency merges.break_cont_carrier_spwithout encountering a return barrier; this PR does not claim to fix it.How to test the change?:
./.claude/commands/build-and-summarize spotlessApply :ddprof-test:testDebug \ '-Ptests=*Frozen*' -Pskip-gtest -PkeepJFRs --max-workers=2vmandvmx: the physical caller is retained but its inline callers are omitted. It passes with original-PC recovery on Linux arm64 JDK 21/27 and macOS arm64 JDK 25/27../.claude/commands/build-and-summarize :ddprof-lib:gtestDebug_frozenDeoptimization_ut.spotlessApplyandgit diff --checkpass. The combined frozen-stack Java regression run passes 18 tests with two conditional GC skips.For Datadog employees: