Skip to content

fix(profiling) wall-time crash after PHP runs on a native thread in ext-grpc - #4198

Draft
realFlowControl wants to merge 3 commits into
masterfrom
florian/grpc-profiler-native-thread-repro
Draft

realFlowControl wants to merge 3 commits into
masterfrom
florian/grpc-profiler-native-thread-repro

Conversation

@realFlowControl

@realFlowControl realFlowControl commented Sep 15, 2026

Copy link
Copy Markdown
Member

Why

The ext-grpc extension may execute PHP code on native threads that are not initialized as PHP threads, meaning they never go through GINIT/GSHUTDOWN.

  • When we collect a stack trace on one of these threads, we store raw string pointers in runtime cache slots and keep the corresponding strings alive in the thread-local CACHED_STRINGS. At this point, everything is fine.
  • The problem starts when that native thread exits. Rust destroys the thread-local CACHED_STRINGS, which frees the strings, but the runtime cache slots still contain pointers to them.
  • Later, when we collect a sample on the main PHP thread for the same function, we see that the runtime cache slot is already initialized and reuse it. That slot now contains a dangling pointer.
  • We dereference it and 💥.

Why did our earlier tests not catch this?
We disable the runtime-cache optimization for CLI, where PHPT's are executed in the CLI SAPI, so we could not see it.

Description

This PR makes the string cache process-global on PHP NTS and keeps it thread-local on ZTS.

It is safe because NTS executes PHP on one thread. It requires unsafe impl Sync because Rust cannot verify that guarantee that there is no concurrent access from another native thread. That would cause undefined behavior. Technically though ext-grpc might cause an access to this from another native thread, just not concurrent.

I am not going to say I am happy with the "fix", but it seems to work.

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

@datadog-official

datadog-official Bot commented Sep 15, 2026

Copy link
Copy Markdown

Pipelines  Tests

Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 12 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | Loader test on arm64 alpine — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-php | Loader test on arm64 libc: [7.4, zts, arm64] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [8.4] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

View all 12 failed jobs.

⚠️ Warnings

2 Test performance regressions detected

Test New execution time Base Execution time Increase DataDog link
profiling/tests/phpt/allocation_generator_01.phpt ([profiling] profiling should not crash during a ZEND_GENERATOR_CREATE)
from php.profiling.tests.phpt
25.59s 614.031881ms +24.98s (+4067%) View in Datadog
profiling/tests/phpt/allocation_generator_01.phpt ([profiling] profiling should not crash during a ZEND_GENERATOR_CREATE)
from PHP.profiling.tests.phpt
24.04s 612.656654ms +23.43s (+3824%) View in Datadog

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 67.46% (-0.01%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7339fc0 | Docs | View more details | Give us feedback!

@realFlowControl realFlowControl changed the title Fix profiler wall-time crash after PHP runs on a native thread fix(profiling) wall-time crash after PHP runs on a native thread in ext-grpc Sep 15, 2026
@github-actions github-actions Bot added the profiling Relates to the Continuous Profiler label Sep 15, 2026
@pr-commenter

pr-commenter Bot commented Sep 15, 2026

Copy link
Copy Markdown

Benchmarks [ profiler ]

Benchmark execution time: 2026-09-17 11:31:07

Comparing candidate commit 7339fc0 in PR branch florian/grpc-profiler-native-thread-repro with baseline commit 2bd2204 in branch master.

Found 0 performance improvements and 2 performance regressions! Performance is the same for 26 metrics, 8 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:php-profiler-timeline-memory-control

  • 🟥 cpu_user_time [+29.945ms; +39.219ms] or [+4.581%; +5.999%]
  • 🟥 execution_time [+32.876ms; +37.441ms] or [+4.692%; +5.343%]

@realFlowControl
realFlowControl force-pushed the florian/grpc-profiler-native-thread-repro branch 2 times, most recently from 0506219 to 8fb0387 Compare September 16, 2026 07:01
@realFlowControl
realFlowControl force-pushed the florian/grpc-profiler-native-thread-repro branch from 8fb0387 to 0dae555 Compare September 16, 2026 08:46
@realFlowControl
realFlowControl force-pushed the florian/grpc-profiler-native-thread-repro branch from efaf119 to 7b5940b Compare September 16, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

profiling Relates to the Continuous Profiler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant