[Bug]: PHP-FPM SIGABRT with SSI and AppSec enabled since 1.25.x

[Yacine58]

Bug report

Summary

When Datadog PHP automatic instrumentation (Single Step Instrumentation / SSI) uses PHP tracer 1.25.x with AppSec enabled, PHP-FPM worker processes may abort with SIGABRT during startup or runtime. The Kubernetes pod and ArgoCD application can nevertheless remain Healthy/Running because the PHP-FPM master process remains alive and replaces the failed workers.

Disabling AppSec resolves the crash while PHP APM traces continue to be emitted.

Impact

The Kubernetes workload remains available and ArgoCD reports it as Healthy/Running. However, PHP-FPM worker processes may abort with SIGABRT, which can cause worker restarts, excessive logs, incomplete APM telemetry, and potentially intermittent request failures.

Environment

• Kubernetes (AWS EKS)
• Datadog Operator / DatadogAgent v2alpha1
• Automatic instrumentation: SSI / Admission Controller
• PHP runtime: PHP-FPM, PHP 8.5.9 NTS
• Zend OPcache: enabled
• Datadog Agent versions tested: 7.81.2 and 7.83.1
• PHP tracer affected: 1.25.x
• Datadog site: datadoghq.eu

Actual behavior

When Datadog PHP automatic instrumentation (Single Step Instrumentation / SSI) uses PHP tracer 1.25.x with AppSec enabled, PHP-FPM worker processes may abort with SIGABRT during startup or runtime. The Kubernetes pod and ArgoCD application can nevertheless remain Healthy/Running because the PHP-FPM master process remains alive and replaces the failed workers.
Observed error:

mutex.cc : RAW: Check w->waitp->cond == nullptr failed:
Mutex::Fer while waiting on Condition

The PHP-FPM process terminates with SIGABRT.

Expected behavior

PHP-FPM pods should start normally when PHP APM and AppSec are both enabled through SSI.

Investigation

The crash was reproduced with Datadog Agent 7.81.2 and 7.83.1, so it does not appear tied to the Agent version.
PHP tracer 1.24.2 was stable in the same environment.
The issue occurs with the PHP tracer 1.25.x line when AppSec is enabled.
PHP-FPM uses a master process that forks workers. We suspect an interaction between this fork model and the sidecar/AppSec communication changes introduced in tracer 1.25.0.

Confirmed workaround

The crash no longer occurs with PHP tracer 1.25.1 when AppSec is disabled through SSI:

With this configuration:

PHP-FPM pods start normally.
PHP APM traces are still produced.
AppSec is disabled for the affected PHP workloads.

Questions

Is this a known regression involving PHP tracer 1.25.x, AppSec, and PHP-FPM?
Is the Mutex::Fer while waiting on Condition SIGABRT related to sidecar/AppSec implementation or fork handling?
Is DD_APPSEC_ENABLED=false the recommended workaround?
Is there a more targeted workaround that preserves AppSec?
Is there a planned fix or version in which AppSec can safely be re-enabled for PHP-FPM?
We can provide sanitized crash logs and DatadogAgent SSI configuration through our existing Datadog Support ticket. ``````

PHP version

8.5.9 NTS

Tracer or profiler version

1.25.1

Installed extensions

dd_library_loader 1.25.1
ddtrace 1.25.1
ddappsec 1.25.1
datadog-profiling 1.25.1
Zend OPcache 8.5.9

Output of phpinfo()

No response

Upgrading from

No response

[cataphract]

Hello. Your failing assertion seems to come abseil, which is not included in any of the extensions you listed. It most likely comes from the grcp extension. It's therefore not clear how ddappsec could be involved, except through some general memory corruption, or maybe by forcing a launch of sidecar -- and therefore a fork() + exec() which would run grpc fork handlers.

Could you please provide the output of phpinfo()? I'd be interested in knowing the full ddtrace and ddappsec configuration (to see whether/how sidecar is launched), whether grpc extension is loaded (grpc.so, the value of its settings, and in particular grpc.enable_fork_support and grpc.poll_strategy).

[Yacine58]

Thank you for pointing us in the direction of the gRPC extension.

We confirmed that the gRPC extension is loaded in the affected PHP-FPM container:

grpc support => enabled
Extension path: /usr/local/lib/php/extensions/no-debug-non-zts-20250925/grpc.so

Initial configuration:

• PHP: 8.5.10 NTS
• PHP tracer: 1.25.1
• ddtrace: 1.25.1
• ddappsec: 1.25.1
• datadog-profiling: 1.25.1
• Zend OPcache: 8.5.10
• grpc.enable_fork_support: 0
• grpc.poll_strategy: no value
• datadog.trace.sidecar_trace_sender: On
• datadog.trace.sidecar_connection_mode: auto
• datadog.trace.forked_process: On
• datadog.appsec.helper_rust_redirection: On

We first observed that setting DD_APPSEC_ENABLED=false prevented the PHP-FPM worker aborts, although repetitive Abseil mutex-related logs were still present.

We then tested the following configuration on a single STG PHP-FPM workload (documentcse), with AppSec enabled:

ddTraceVersions:
php: "1.25.1"
ddTraceConfigs:

• name: "DD_APPSEC_ENABLED"
  value: "true"

We also added the following PHP configuration without rebuilding the image:

grpc.enable_fork_support=1

The effective configuration inside the container was confirmed:

datadog.appsec.enabled => On => On
grpc.enable_fork_support => 1 => 1
GRPC_ENABLE_FORK_SUPPORT => 1

After restarting the service with AppSec enabled and grpc.enable_fork_support=1, the frequency of the following logs decreased significantly:

[mutex.cc : 2445] RAW: Check w->waitp->cv_word == nullptr failed:
Mutex::Fer with pending CondVar queueing

However, the messages did not disappear completely. Occasional occurrences are still visible.

The Kubernetes pod and the ArgoCD application remained Healthy/Running during the test. This is consistent with the issue affecting PHP-FPM worker processes while the PHP-FPM master process remains alive and replaces failed workers.

Our current conclusion is that grpc.enable_fork_support=1 provides a significant partial mitigation, but does not completely resolve the issue. The remaining assertions may still involve the interaction between PHP-FPM fork handling, gRPC, and the Datadog tracer/AppSec sidecar.

We have not changed grpc.poll_strategy; it remains unset (no value).

Could you please advise:

1. Is grpc.enable_fork_support=1 the recommended configuration for PHP-FPM applications using gRPC?
2. Should we test a specific value for grpc.poll_strategy?
3. Could the remaining assertions be caused by gRPC fork handling, or could the Datadog sidecar/AppSec process still be involved?
4. Is there a Datadog configuration that prevents the sidecar from being launched before the PHP-FPM fork?
5. Would a stack trace or core dump help identify whether the failing Abseil mutex belongs to gRPC or to the Datadog sidecar?

We can provide the complete sanitized phpinfo() output, additional logs, and the Kubernetes/SSI configuration if needed.

[cataphract]

The documentation status that you should enable grpc.enable_fork_support if you use "pcntl_fork() and related functions". This installs atfork handlers. Simplifying a lot, they lock/flush some state before the fork, unlock afterwards in the parent and resets all state in the child so it cannot interfere with the parent's transports (which is why you can get "channel closed" errors in the child if you use stale php objects).

So that is the direct answer to your question. Do you use forking in your application? If so, what do you do after the fork? Also, I'm assuming that your're running only a gRPC client. Running a server (Grpc\Server) is not recommended in production and starting a server and forking (rather than starting a server after the fork) is not supported.

Now on the other points:

Should we test a specific value for grpc.poll_strategy?

You should probably leave the default. The only reason I asked was that the atfork handlers don't support all poll strategies.

Could the remaining assertions be caused by gRPC fork handling, or could the Datadog sidecar/AppSec process still be involved?

At this point, I don't see a lot of evidence for the involvement of Datadog sidecar/AppSec. You already said the problem persists if appsec is disabled and your configuration is such that sidecar would be launched even in the absence of appsec. You could also try disabling the tracer and profiling to see if the error persists even then.

Is there a Datadog configuration that prevents the sidecar from being launched before the PHP-FPM fork?

Sidecar is not launched before the fork. It's lazily launched the first time it's needed, in one of the children (workers).

Would a stack trace or core dump help identify whether the failing Abseil mutex belongs to gRPC or to the Datadog sidecar?

It definitely belongs to grpc. A stack trace might help, but my hopes are not very high, because it'll probably be difficult to link the assertion failure to the earlier cause of the corrupted state. If you could provide one, try to get the stacktrace of all the threads.
A coredump is a bit more likely to help, but since it contains the contents of the process memory, it's likely to have sensitive data. We would also need the exact binaries involved in the process.

[cataphract]

Ah, and yes, a sanitized phpinfo() would help. For instance, if you have an opache preload script, ensure it's not using grpc. Because that happens pre-fork, in the fpm master process.

[bwoebi]

@Yacine58 Did you already have a chance to look into it?