Skip to content

fix(live-debugger): fix use-after-free, tags leak, and Windows 7.x crash on probe removal#4036

Merged
Leiyks merged 6 commits into
masterfrom
leiyks/fix-live-debugger-shared-probe-id
Jul 16, 2026
Merged

fix(live-debugger): fix use-after-free, tags leak, and Windows 7.x crash on probe removal#4036
Leiyks merged 6 commits into
masterfrom
leiyks/fix-live-debugger-shared-probe-id

Conversation

@Leiyks

@Leiyks Leiyks commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Fixes live-debugger probe-lifetime bugs surfaced by a flaky LEAKED on debugger_enable_dynamic_config.phpt, plus a Windows-only crash the new regression test exposed.

Fixes

  1. Use-after-free (original failure). spans_map was keyed inconsistently — probe.id on apply/remove vs config_id on the DI-reinstall path. Two configs sharing a probe.id orphaned a hook whose parsed-config box was then freed, so its next fire read the freed probe.id. Now keyed by config_id everywhere. Reproduced on PHP 7.1 & 8.3 under valgrind.
  2. tags leak. probe.into() allocates a CharSliceVec for tags that dd_probe_dtor never freed. Added ddog_drop_probe (consumes the FFI probe by value) so drop glue frees tags + nested allocations in one place.
  3. PHP 7.x Windows crash (0xC0000005). MSVC 2017 if-converted the inline ternary building zai_hook_remove's scope arg into an unconditional ZSTR_LEN(def->scope), a NULL deref for global functions. Build the zai_str views with explicit if-guards. PHP 8.x (MSVC 2019) / Linux unaffected.

Also drops the stale active_live_debugger_hooks entry on removal so a later lookup of the same id can't use-after-free the freed def.

Test & verification

tests/ext/live-debugger/debugger_remove_shared_probe_id.phpt installs two configs sharing a probe.id (distinct tags), removes both without firing, then calls the target — exercising the UAF, the tags allocation, and the Windows removal path. Green across the full test_c matrix: windows 7.2–8.5 and ASAN (Linux) 7.4–8.5; valgrind clean (0 bytes lost, no double-free).

…use-after-free

The live debugger tracks installed probe hooks in `spans_map`, but keyed it
inconsistently: apply_config and remove_config used the probe's `id` field,
while the DI-enable reinstall path used the `active` map key (the config_id).

`active` is keyed by config_id, which is unique per remote-config path, whereas
probe.id is not: two distinct configs can carry the same probe id. When they do
(or when a probe is installed via different paths), the second install overwrites
the probe.id-keyed spans_map entry, orphaning the first hook. Removing a config
then tears down the wrong hook (or none) and frees the parsed-config box while an
orphaned hook still borrows its strings. The orphan's first fire afterwards reads
the freed probe id in ddog_debugger_diagnostics_create_unboxed -> use-after-free.

Only valgrind observes it (plain runs read freed-but-intact bytes), which is why
it surfaced intermittently as a LEAKED test in test_extension_ci.

Key spans_map by config_id everywhere (apply_config, remove_config, and the
already-correct reinstall path) so every config's hook is tracked and removed
independently. Adds tests/ext/live-debugger/debugger_remove_shared_probe_id.phpt,
which reproduces the UAF (two configs sharing a probe id) and is clean with the fix.

Verified under valgrind on PHP 7.1 and 8.3: the reproducer leaks before the fix
and passes after; the whole live-debugger suite is clean.
@Leiyks
Leiyks requested review from a team as code owners July 8, 2026 13:02
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 8, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 3 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | min install tests   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [8.4]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [7.0]   View in Datadog   GitLab

❄️ 1 New flaky test detected

tmp/build_extension/tests/ext/live-debugger/debugger_remove_shared_probe_id.phpt (Removing live debugger configs that share a probe id must not leave a dangling hook) from PHP.tmp.build_extension.tests.ext.live.debugger   View in Datadog
001+ DDBC:60-alter_di_config-entry
001- string(7) "removed"
002+ DDBC:01-set_probe-entry
003+ DDBC:02-set_span_probe-entry
004+ DDBC:03-init-entry
005+ DDBC:04-init-before-hook_install
006+ DDBC:05-init-after-hook_install id=189
007+ DDBC:06-init-before-map_add id=189
008+ DDBC:07-init-map_added id=189
009+ DDBC:01-set_probe-entry
...

New test introduced in this PR is flaky.

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 54.08% (-0.03%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f4b2785 | Docs | Datadog PR Page | Give us feedback!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d4ab496dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread components-rs/remote_config.rs
Comment thread tests/ext/live-debugger/debugger_remove_shared_probe_id.phpt Outdated
Leiyks added 2 commits July 8, 2026 15:59
The regression test used a distinct `tags` entry per config only to give the two
configs different remote-config paths. But a non-empty `tags` makes probe.into()
allocate an FFI CharSliceVec that the C side never frees (a separate latent leak
for tagged probes), which LeakSanitizer flags in the ASAN 'multiple observers'
job (32 bytes / 2 objects) and breaks the test's expected output.

Use an ignored unknown field instead: same two distinct config paths, same probe
id, no per-config allocation -> no unrelated LSan noise. Verified under valgrind
(clean) and the two configs still install at distinct paths with the shared id.
probe.into() heap-allocates a CharSliceVec for the probe `tags`, but the C side
(def->probe) only freed the nested span-decoration / log allocations in
dd_probe_dtor -- never the tags vec -- so every probe carrying tags leaked it
(LeakSanitizer: 32 bytes / 2 objects in the ASAN 'multiple observers' job).

Add ddog_drop_probe, which consumes the FFI probe by value so its drop glue
frees the tags CharSliceVec together with the nested span-decoration / log
allocations, and call it from dd_probe_dtor in place of the piecemeal drops.
Cleanup is now a single, consistent operation covering every FFI-owned field.

Restore the shared-probe-id regression test to distinguish its two configs by
`tags`, so it also exercises this path (a tags leak would resurface under LSan).

Verified on PHP 8.3: the live-debugger suite is clean under valgrind (no UAF,
no double-free), and valgrind --leak-check=full on a tagged probe shows 0 bytes
definitely lost with no CharSliceVec leak record.
@Leiyks Leiyks changed the title fix(live-debugger): key spans_map by config_id to prevent probe hook use-after-free fix(live-debugger): fix probe hook use-after-free and tags leak on config removal Jul 8, 2026
…; trim comments

Address Codex review:
- apply_config: remove any hook already installed for a config id before
  installing the replacement, so an in-place Add (Occupied entry) can't orphan
  the previous hook into the dropped parsed config.
- regression test: await both colliding probes (2) before removing, so it
  actually exercises the shared-probe-id collision.

Also condense the comments added in this PR.
@pr-commenter

pr-commenter Bot commented Jul 8, 2026

Copy link
Copy Markdown

Benchmarks [ tracer ]

Benchmark execution time: 2026-07-14 03:15:00

Comparing candidate commit f4b2785 in PR branch leiyks/fix-live-debugger-shared-probe-id with baseline commit 024fe4a in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 193 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:MessagePackSerializationBench/benchMessagePackSerialization

  • 🟩 execution_time [-3.677µs; -2.883µs] or [-3.540%; -2.775%]

@gh-worker-ownership-write-b05516
gh-worker-ownership-write-b05516 Bot removed the request for review from a team July 9, 2026 15:14
@Leiyks
Leiyks force-pushed the leiyks/fix-live-debugger-shared-probe-id branch from f728dc3 to cef3cd0 Compare July 13, 2026 12:57
@Leiyks
Leiyks force-pushed the leiyks/fix-live-debugger-shared-probe-id branch 2 times, most recently from 8fdcc12 to 58f16aa Compare July 14, 2026 01:44
@Leiyks Leiyks changed the title fix(live-debugger): fix probe hook use-after-free and tags leak on config removal fix(live-debugger): fix use-after-free, tags leak, and Windows 7.x crash on probe removal Jul 14, 2026
…map entry

The regression test exposed a PHP 7.x Windows-only crash (0xC0000005) in
dd_remove_live_debugger_probe: MSVC 2017 if-converts the inline ternary that
builds zai_hook_remove's scope/function args into an unconditional ZSTR_LEN
load, a NULL deref for global functions (scope == NULL). Build the zai_str
views with explicit if-guards, as the install path already does. PHP 8.x
(MSVC 2019) and Linux were unaffected.

Also drop the stale active_live_debugger_hooks entry on removal so a later
lookup of the same id can't use-after-free the already-freed def.
@Leiyks
Leiyks force-pushed the leiyks/fix-live-debugger-shared-probe-id branch from 58f16aa to f4b2785 Compare July 14, 2026 01:56

@bwoebi bwoebi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still not persuaded that probe_ids can effectively be shared, but it won't break anything. So let's do that.

@Leiyks
Leiyks merged commit 17a0855 into master Jul 16, 2026
2154 of 2158 checks passed
@Leiyks
Leiyks deleted the leiyks/fix-live-debugger-shared-probe-id branch July 16, 2026 12:32
@github-actions github-actions Bot added this to the 1.23.0 milestone Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants