Skip to content

Traces do not appear in Datadog Web UI when AOT Cache is enabled, #10479

Closed
#12030
Closed
Traces do not appear in Datadog Web UI when AOT Cache is enabled,#10479
#12030
Assignees
mcculls
Labels
type: bug reportBug report
Milestone
1.68.0

Description

@footaku
footaku
opened on Jan 30, 2026

Tracer Version(s)

1.58.2(, 1.57.0, 1.58.0 too)

Java Version(s)

25

JVM Vendor

Eclipse Adoptium / Temurin

Bug Report

Behavior

When the application is started with AOT Cache enabled, traces do not appear in Datadog Web UI.
Using the same container image and the same JAR, traces do appear if AOT Cache is disabled.
Also, if AOT Cache is enabled but AOT Class Linking is disabled, traces do appear.

With DD_TRACE_DEBUG=true, the following log appears much more often in the NG case than in the OK cases, but it is unclear if this is the root cause:
DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=xxx.xxx.Xxx instrumentation.target.context=yyy.yyy.Yyy->zzz.zzz.Zzz

From the logs, sending to the Datadog Agent seems successful:

[dd-trace-processor] EXCLUDE_TELEMETRY datadog.trace.agent.common.writer.ddagent.DDAgentApi - Success while sending 8 (size=6KB) traces. Total: 18, Received: 18, Sent: 18, Failed: 0.
[dd-trace-processor] DEBUG datadog.trace.agent.common.sampling.RateByServiceTraceSampler - Update service sampler rates: http://dd-agent:8126/v0.4/traces -> {rate_by_service={service:,env:=1.0}}
[dd-trace-processor] DEBUG datadog.trace.agent.common.writer.PayloadDispatcherImpl - Successfully sent 8 traces to the API

I would appreciate any guidance on potential causes or possible workarounds.

Expected Behavior

Traces appear in Datadog Web UI

Reproduction Code

Case 1 (NG)

The application starts, but traces do not appear.

java -javaagent:/dd-java-agent.jar \
  -Xlog:aot \
  -XX:AOTCache=/workspace/application.aot \
  -jar application.jar
`Failed to add context-store field` output 
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyStatement instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyLeakTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=software.amazon.jdbc.util.monitoring.AbstractMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$KeepaliveTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$MaxLifetimeTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$HouseKeeper instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.SocketWrapperBase$OperationState instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.Acceptor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessorMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.backpressure.BackpressureMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.PreviousSessionFinalizer instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.MovePreviousSession instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=tools.jackson.databind.util.internal.PrivateMaxEntriesMap$AddTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=ch.qos.logback.classic.spi.LoggingEvent instrumentation.target.context=ch.qos.logback.classic.spi.ILoggingEvent->datadog.trace.bootstrap.instrumentation.api.AgentSpanContext
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.StatementImpl instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.springframework.beans.factory.support.DisposableBeanAdapter instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.ConnectionImpl instrumentation.target.context=java.sql.Connection->datadog.trace.bootstrap.instrumentation.jdbc.DBInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.AbandonedConnectionCleanupThread instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyConnection instrumentation.target.context=java.sql.Connection->datadog.trace.bootstrap.instrumentation.jdbc.DBInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.common.SequentialScheduler$SchedulableTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.Http1Exchange$Http1Publisher$WriteTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=sun.rmi.transport.tcp.TCPTransport$AcceptLoop instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.CompletableFuture$UniCompletion instrumentation.target.context=java.util.concurrent.CompletableFuture$UniCompletion->datadog.trace.bootstrap.instrumentation.java.concurrent.ConcurrentState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.FutureTask instrumentation.target.context=java.util.concurrent.RunnableFuture->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor instrumentation.target.context=java.util.concurrent.ThreadPoolExecutor->java.lang.Boolean
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.net.HttpURLConnection instrumentation.target.context=java.net.HttpURLConnection->datadog.trace.bootstrap.instrumentation.httpurlconnection.HttpUrlState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ForkJoinTask instrumentation.target.context=java.util.concurrent.ForkJoinTask->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyStatement instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.ProxyLeakTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=software.amazon.jdbc.util.monitoring.AbstractMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$KeepaliveTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$MaxLifetimeTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.zaxxer.hikari.pool.HikariPool$HouseKeeper instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.SocketWrapperBase$OperationState instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.tomcat.util.net.Acceptor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessorMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.backpressure.BackpressureMonitor instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.PreviousSessionFinalizer instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=io.sentry.MovePreviousSession instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=tools.jackson.databind.util.internal.PrivateMaxEntriesMap$AddTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=ch.qos.logback.classic.spi.LoggingEvent instrumentation.target.context=ch.qos.logback.classic.spi.ILoggingEvent->datadog.trace.bootstrap.instrumentation.api.AgentSpanContext
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.StatementImpl instrumentation.target.context=java.sql.Statement->datadog.trace.bootstrap.instrumentation.jdbc.DBQueryInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=org.springframework.beans.factory.support.DisposableBeanAdapter instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.ConnectionImpl instrumentation.target.context=java.sql.Connection->datadog.trace.bootstrap.instrumentation.jdbc.DBInfo
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=com.mysql.cj.jdbc.AbandonedConnectionCleanupThread instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.common.SequentialScheduler$SchedulableTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=jdk.internal.net.http.Http1Exchange$Http1Publisher$WriteTask instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=sun.rmi.transport.tcp.TCPTransport$AcceptLoop instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.CompletableFuture$UniCompletion instrumentation.target.context=java.util.concurrent.CompletableFuture$UniCompletion->datadog.trace.bootstrap.instrumentation.java.concurrent.ConcurrentState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.FutureTask instrumentation.target.context=java.util.concurrent.RunnableFuture->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor$Worker instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor instrumentation.target.context=java.util.concurrent.ThreadPoolExecutor->java.lang.Boolean
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.net.HttpURLConnection instrumentation.target.context=java.net.HttpURLConnection->datadog.trace.bootstrap.instrumentation.httpurlconnection.HttpUrlState
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.util.concurrent.ForkJoinTask instrumentation.target.context=java.util.concurrent.ForkJoinTask->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope

Case 2 (OK)

Traces appear when AOT Cache is not specified.

java -javaagent:/dd-java-agent.jar \
  -Xlog:aot \
  -jar application.jar
`Failed to add context-store field` output 
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope

Case 3 (OK)

Traces appear when AOT Class Linking is disabled.

java -javaagent:/dd-java-agent.jar \
  -Xlog:aot \
  -XX:-AOTClassLinking \
  -XX:AOTCache=/workspace/application.aot \
  -jar application.jar
`Failed to add context-store field` output 
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope
[main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextMatcher - Failed to add context-store field - instrumentation.target.class=java.lang.VirtualThread instrumentation.target.context=java.lang.VirtualThread->datadog.trace.bootstrap.instrumentation.api.AgentScope

Added

Environment

  • Datadog
    • dd-trace-java 1.58.2 / Agent 7.73.3
  • Kubernetes (Amazon EKS)
  • Eclipse Temurin 25 
    java --version
openjdk 25 2025-09-16 LTS
OpenJDK Runtime Environment Temurin-25+36 (build 25+36-LTS)
OpenJDK 64-Bit Server VM Temurin-25+36 (build 25+36-LTS, mixed mode, sharing)
  • Spring Boot 4.0.0
    • hikariCP
    • MySQL (Amazon Aurora)

Activity

  1. footaku
    added
    type: bug fixBug fix
    on Jan 30, 2026

  2. mcculls commented on Jan 30, 2026

    @mcculls
    mcculls
    on Jan 30, 2026
    Contributor

    Hi @footaku - the -XX:AOT option is not recognized by the Eclipse/Temurin JVM, I assume you're using IBM/Semeru? Could you post the full version of the JVM to be sure?

    I've confirmed that traces are received when using Eclipse/Temurin 25 with -XX:AOTCache so this is probably something specific to IBM Semeru's AOT, which is different to OpenJDK's.

    To help continue the investigation could you post the exact switches you used to create the AOT cache?

  3. footaku commented on Jan 30, 2026

    @footaku
    footaku
    on Jan 30, 2026
    Author

    Hi @mcculls - Thanks for reaching out.
    And sorry — the command I wrote was wrong.
    The -XX:AOT part should have been -Xlog:aot. I have updated the issue description as well.

    This is the command I use to create the AOT cache:

    export JDK_JAVA_OPTIONS="-javaagent:dd-java-agent.jar \
-Xmx256m \
-Xms256m \
-Dspring.context.exit=onRefresh \
-Xlog:aot \
-XX:AOTCacheOutput=/workspace/application.aot"
java -jar application.jar

    Alternatively, I also create a class to terminate the application and use a method like this:

    import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.ApplicationContext;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

@Component
@ConditionalOnProperty(prefix = "aot.training", name = "enabled", havingValue = "true")
public class AOTTrainingTerminator {
  private final Logger logger = LoggerFactory.getLogger(AOTTrainingTerminator.class);
  private final ApplicationContext context;

  public AOTTrainingTerminator(ApplicationContext context) {
    this.context = context;
  }

  @EventListener(ApplicationReadyEvent.class)
  public void terminate() {
    logger.info("Terminating the AOT training.");
    System.exit(SpringApplication.exit(context, () -> 0));
  }
}
    export AOT_TRAINING_ENABLED=true
export JDK_JAVA_OPTIONS="-javaagent:dd-java-agent.jar \
-Xmx256m \
-Xms256m \
-Xlog:aot \
-XX:AOTCacheOutput=/workspace/application.aot"
java -jar application.jar

  4. mcculls commented on Jan 30, 2026

    @mcculls
    mcculls
    on Jan 30, 2026
    Contributor

    Hi @footaku - can you post the full version string of your JVM, thanks

  5. footaku commented on Jan 30, 2026

    @footaku
    footaku
    on Jan 30, 2026
    Author

    Hi @mcculls - this is the version string of the JVM:

    java --version
openjdk 25 2025-09-16 LTS
OpenJDK Runtime Environment Temurin-25+36 (build 25+36-LTS)
OpenJDK 64-Bit Server VM Temurin-25+36 (build 25+36-LTS, mixed mode, sharing)
  6. mcculls
    self-assigned this
    on Mar 5, 2026

  7. davidgoldtempo commented on Mar 18, 2026

    @davidgoldtempo
    davidgoldtempo
    on Mar 18, 2026

    We have a similar set up and encounter the same problem. Without -XX:-AOTClassLinking we get a 60% improvement in startup resource usage, but the profiling data doesn't make it into Datadog properly (we notice the dd tags are missing). With -XX:-AOTClassLinking our improvement drops to 10%, but our traces appear correctly.

  8. snuderl commented on Apr 21, 2026

    @snuderl
    snuderl
    on Apr 21, 2026

    we are similarly seeing issues. In our cases its not all traces but just subset of them missing. Seems to be mainly automatically instrumented things, while manual trace usage works

  9. vigenere23 commented on May 22, 2026

    @vigenere23
    vigenere23
    on May 22, 2026

    Can also confirm the bug. This is preventing us from using AOT, which is unfortunate.

  10. snuderl commented on Jun 9, 2026

    @snuderl
    snuderl
    on Jun 9, 2026

    @mccu any update on this?

  11. mcculls commented on Jun 9, 2026

    @mcculls
    mcculls
    on Jun 9, 2026
    Contributor

    @snuderl @vigenere23 can you post your JVM details?

    If you're able to share a buildable example that demonstrates the issue that would really help progress this issue and ensure any fix will also apply to your setup - so far simple examples such as Spring petclinic don't exhibit this issue.

  12. snuderl commented on Jun 11, 2026

    @snuderl
    snuderl
    on Jun 11, 2026

    Based on the agent feedback the reason petclinic does not reproduce is because it uses synchronous MVC which partially works.

    Anyway we are seeing missing stuff in atleast 3 places:

    • kafka traces
    • http
    • scheduled tasks in spring

    I have included this patch against petclinic that shows all 3 of the issues.

    petclinic-aot-reproducer.patch Note also it does not seem to be related to specific jvm distribution (I tried a few)

    # 1. Clone petclinic
git clone https://github.com/spring-projects/spring-petclinic.git
cd spring-petclinic
 
# 2. Apply patch
git apply petclinic-aot-reproducer.patch
 
# 3. Download prerequisites
wget https://corretto.aws/downloads/latest/amazon-corretto-25-x64-linux-jdk.tar.gz
tar xzf amazon-corretto-25-x64-linux-jdk.tar.gz
export JAVA_HOME=$(pwd)/amazon-corretto-25.0.3.9.1-linux-x64
 
wget -O dd-java-agent.jar https://dtdg.co/latest-java-tracer
export DD_AGENT=$(pwd)/dd-java-agent.jar
 
# 4. Run
bash test-aot.sh

    which gives me the following output:

    === SUMMARY ===
  classlink:                 12 failures (6 unique)
  noclasslink:                2 failures (1 unique)
  noaot:                      2 failures (1 unique)

  13. mcculls commented on Jun 11, 2026

    @mcculls
    mcculls
    on Jun 11, 2026
    Contributor

    thanks @snuderl!

  14. davidgoldtempo commented on Jun 11, 2026

    @davidgoldtempo
    davidgoldtempo
    on Jun 11, 2026

    snuderl's repro worked for me. We have this problem with synchronous MVC, it appears somehow related to jib with containerizingMode=packaged* rather than the fat jar. When trying that out the 12 failures turn into 90+ including JDBC/Hibernate/HikariCP/CoyoteAdapter/DispatcherServlet.

  15. mcculls commented on Jun 26, 2026

    @mcculls
    mcculls
    on Jun 26, 2026
    Contributor

    I'm still looking into the trace differences but wanted to mention that the Failed to add context-store field messages are expected when using AOT.

    That message means the tracer wasn't able to inject a field into the class. This can happen when classes are materialized straight from the AOT cache because they bypass the usual JVM instrumentation layer. Once the class is loaded the JVM doesn't let you add a field to the class. Basically you cannot change the shape of the class, but you can change what a method does - so you can still apply entry-exit advice.

    Note this shouldn't lead to missing traces because the instrumentation will automatically fall back to the global weak-map to track context - so something else is going on that does affect traces in certain circumstances.

  16. 2 remaining items

  17. PerfectSlayer
    added
    type: bug reportBug report
    and removed
    type: bug fixBug fix
    on Jul 6, 2026
  18. CharlyRien
    added a commit that references this issue on Jul 22, 2026

    Fix async context loss when field injection is unavailable

    78f023d

  19. Meemaw commented on Jul 30, 2026

    @Meemaw
    Meemaw
    on Jul 30, 2026
    Contributor

    Any update on this @mcculls?

  20. mcculls commented on Jul 31, 2026

    @mcculls
    mcculls
    on Jul 31, 2026
    Contributor

    Hi @Meemaw - I now have a weak-map approach that avoids any dropped context (0% loss) while still respecting an upper bound.

    The instrumentation fix is already in place but the new weak-map won't make it into 1.65.0 (too close to the scheduled release to safely get it in) - I do expect it to be part of 1.66.0 expected at the start of September.

  21. gibkim commented on Sep 16, 2026

    @gibkim
    gibkim
    on Sep 16, 2026

    TL;DR: we now have all spans collected with the AOT cache still enabled — the fix was disabling a single profiler feature: -Ddd.profiling.queueing.time.enabled=false.

    I hit the same symptom (JDK 25 + AOT cache + jib containerizingMode=packaged: tracer starts, agent_error: false, but zero spans) and traced it to a different mechanism than the context-store fallback discussed above.

    Reproduced with both dd-java-agent 1.65.1 and 1.66.0 — identical zero-span behaviour, so this is not a recent regression in the agent.

    UnwrappingVisitor (queueing-time profiling) adds the TaskWrapper interface — a structural change the JVM rejects on retransformation. With an AOT cache the affected classes are materialized before premain, so the batch fails, and since there is no BatchAllocator configured, RedefinitionStrategy aborts the whole install. That would also explain why @davidgoldtempo saw failures jump to 90+ with jib packaged: more classes land in that single doomed batch.

    I've opened #12506 which retries the failed batch without the interface change, so instrumentation survives. Queueing-time profiling keeps working too — only the task field's type resolution is lost (wrapper class instead of the real one; measured with JFR in #12506), instead of the feature being disabled outright as the flag above does.

  22. mcculls commented on Sep 16, 2026

    @mcculls
    mcculls
    on Sep 16, 2026
    Contributor

    Hi @gibkim - thanks for the detail, but your particular issue is unrelated to this one (to confirm I tried your patch from #12506 with the reproducer in https://gist.github.com/CharlyRien/b50510ace2659c3d2d68abbd5dc34e11 and it didn't help, neither did using the -Ddd.profiling.queueing.time.enabled=false switch)

  23. mcculls commented on Sep 16, 2026

    @mcculls
    mcculls
    on Sep 16, 2026
    Contributor

    Hi @CharlyRien @Meemaw v1.66.0 shipped with a feature to help with the loss of context under load on AOT.

    To use this mode set the following environment variable:

    DD_TRACE_RUNTIME_CONTEXT_MAP_PER_STORE=false

    or add this JVM option:

    -Ddd.trace.runtime.context.map-per-store=false

    The next release will include some performance improvements for this mode and hopefully start rolling it out by default. (we'll most likely do this gradually based on the Java version being used.)

  24. CharlyRien commented on Sep 16, 2026

    @CharlyRien
    CharlyRien
    on Sep 16, 2026

    Thanks @mcculls, I confirm that the fix works well, we are testing it in pre-prod for now, and will deploy it soon on our java services fleet.

    Thanks again.

  25. davidgoldtempo commented on Sep 22, 2026

    @davidgoldtempo
    davidgoldtempo
    on Sep 22, 2026

    Upon testing out with DD_TRACE_RUNTIME_CONTEXT_MAP_PER_STORE=false we found traces were still dropped. Notably it's only when we have profiling enabled, which I could be wrong about but I believe that was the case for us before version 1.66.

    Here's an attempt to repro the problem, including a run with the flag @gibkim mentioned above. Sorry for the vibe coding, hope it is helpful. https://gist.github.com/davidgoldtempo/312b18e8790f5afe4c0913e34f96e613

  26. mcculls commented on Sep 22, 2026

    @mcculls
    mcculls
    on Sep 22, 2026
    Contributor

    Hi @davidgoldtempo if it's only when profiling is enabled then it will be related to #12540 (which has a different root cause to the context store issue, even though the resulting effect is the same)

    Thanks for the repro

  27. mcculls commented on Oct 23, 2026

    @mcculls
    mcculls
    on Oct 23, 2026
    Contributor

    👋 quick note to say that the v1.68.0 release (expected next week) will use the new context store approach for Java 25+

    With this change you shouldn't have to explicitly set DD_TRACE_RUNTIME_CONTEXT_MAP_PER_STORE=false when using AOT on Java 25+

  28. mcculls
    added this to the 1.68.0 milestone on Oct 23, 2026

  29. dd-octo-sts commented on Oct 26, 2026

    @dd-octo-sts
    dd-octo-stsBot
    on Oct 26, 2026 – with DD Octo STS
    Contributor

    🤖 This issue has been addressed in the latest release. Check full details in the Release Notes.

  30. dd-octo-sts
    closed this as completedon Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

type: bug reportBug report

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions