test(warmup): Add warm-up tests that auto-cover the generated provider of every service module

[joviegas]

Motivation and Context

CRaC warm-up codegen emits a <Service>WarmUpProvider for each service module, but no test ran those generated providers. The existing warmup-tests module covered the SdkWarmUp orchestrator and HTTP-client warm-up with hand-written providers.

This change adds tests that run the generated provider of every service. The tests put all services on the classpath through the aws-sdk-java aggregate artifact. The aggregate picks up each new service, so the tests cover a new service with no change to this module. A completeness test turns a new service that ships without a registered provider into a build failure instead of a silent gap.

Modifications

Added to test/warmup-tests:

• AllServicesWarmUpTest: parameterized over every generated provider that ServiceLoader finds, one sync case and one async case per provider. Each case asserts that warm-up completes without throwing, that the selected warm-up operation runs (OperationRecordingInterceptor records it), and that SDK loggers emit no warn or error logs (LogCaptor captures them, filtered to SDK loggers). KNOWN_NO_OP_PROVIDERS lists services whose selector picked no operation because every operation is streaming, event-streaming, or deprecated. The test asserts these providers record no operation, and an entry fails once codegen starts selecting an operation for that service.

• SdkWarmUpPrimeAllServicesTest: runs the customer entry point SdkWarmUp.prime() with all services on the classpath and asserts it logs no warm-up failure for any generated service provider. The test scopes the assertion to service-provider warnings because the HTTP-client warm-up step logs the same message when an HTTP client fails to build.

• WarmUpProviderCompletenessTest: scans the classpath jars and asserts every service jar registers a SdkWarmUpProvider in META-INF/services. The ServiceLoader-driven tests generate no case for an unregistered service, so this test reads the jars directly. The test enumerates jars with ClassLoader.getResources instead of java.class.path because surefire runs the test JVM with a manifest-only booter jar.

• OperationRecordingInterceptor and its execution.interceptors registration: records the name of the operation that warm-up invokes, so the tests confirm the selected operation ran.

• pom.xml: adds test-scope dependencies on the aws-sdk-java aggregate (all services on the classpath), test-utils (LogCaptor), the log4j2 binding (log4j-core, log4j-slf4j-impl), and http-auth-aws-crt (sigv4a signer, required by services whose endpoint resolves to sigv4a). Disables JaCoCo for this module: it loads every service client, and instrumenting the two largest generated clients (SSM, Glue) pushes them past the JVM class-size limit.

Testing

License

• I confirm that this pull request can be released under the Apache 2 license

[zoewangg]

Is this test really necessary? What potential issue would this test catch?

[joviegas]

This test catches a service jar that ships without a valid SdkWarmUpProvider registration in META-INF/services. The other two tests discover providers through ServiceLoader, and ServiceLoader only sees registered providers, so an unregistered service simply produces no test case and the suite passes while that service silently loses warm-up coverage. This test reads the classpath jars directly, so the same defect becomes a build failure.

This can happen because codegen writes the registration as a plain text file (WarmUpProviderRegistrationTask). A codegen regression can skip or corrupt that file if there is customized code for a module and without any unit test failing, and a hand-written META-INF/services file added to a module later would overwrite the generated one during packaging. This test confirms every service jar has its registration, and that is what gives the other two tests their input: they only execute providers that are registered.

[zoewangg]

Hmm, this feels like a bit overkill IMO. I can't think of a case where we'd miss generating files for certain services by mistake. I wonder if this would introduce more work, for example, we need to update this test file whenever we need to skip certain services intentionally.

[joviegas]

Thanks for the feedback. Quick clarification: do you mean just the loading test (WarmUpProviderCompletenessTest), or all the tests here as well?

Their main purpose is regression prevention: making sure every module registers its META-INF service entry and that it isn't accidentally overwritten for a module later.

Happy to discuss a lighter alternative offline, and I'm fine removing it if we agree it isn't needed.

[zoewangg]

I meant WarmUpProviderCompletenessTest (basically any tests testing implementation details instead of behavior). What's the difference between SdkWarmUpPrimeAllServicesTest and AllServicesWarmUpTest? Can we consolidate them?

[joviegas]

Agreed on both points.
On WarmUpProviderCompletenessTest: Removed it now
On consolidation: yes, good point, we can merge them. The two classes overlapped, they just checked different layers:

• SdkWarmUpPrimeAllServicesTest exercised the customer entry point SdkWarmUp.prime() once with every service on the classpath and asserted no provider logs a warm-up failure.
• AllServicesWarmUpTest exercised each generated provider individually and asserted the warm-up actually invoked its selected operation and emitted no SDK warnings.

I've consolidated them into a single AllServicesWarmUpTest that drives each client through the public API SdkWarmUp.prime(Client.class) (sync and async), asserting the operation is invoked with no WARN logged, and keeps one SdkWarmUp.prime() test that warms all services at once. So we now test only through public APIs.

[zoewangg]

nit: technically, this feature/test doesn't directly interact with CRaC, let's do a scan and remove the wording. Can be done in a separate PR

[joviegas]

Removed from CRaC from here will do the scan and remove remaining with another PR.

[zoewangg]

Question: any reason we want to run tests for all services? How long does it take to run all tests?

[joviegas]

Yes, we run all services because the warm-up operation is selected per service from each service's model; only executing every generated provider proves each one works. The full module runs offline (Canned Req/Response).

These test guard the customer path: a customer with all services on the classpath calls SdkWarmUp.prime(), and a broken provider logs a WARN and skips that service's warm-up. These tests turn that customer-visible WARN into a build failure here.

New services are covered automatically, the aggregate dependency picks up each new service, and the completeness test fails the build if a service ships without a registered provider

[joviegas]

I will get the exact time once I merge the PR 7183. The issues mentioned in this PR were caught by these tests

[joviegas]

The building of all the modules without mvn -T1C maven option takes 50 mins.
The junits overall take 1 minute.
The overall codebuild takes 94m if executed as part of exsiting jdk Junit buildss (That was because a a failed build performed a retry)
Thus I am planning to move this as a separate test similar to v2-migration-test and s3-regression test that can run parallely.

I tested this on codebuild with the below mvn command similar to v2-migration-test

mvn clean install -pl :warmup-tests,:bom-internal -am -P quick -T1C $MAVEN_OPTIONS
mvn install -pl :warmup-tests -T2C $MAVEN_OPTIONS

This took 23.9 mins

[joviegas]

Apologies, the earlier 94 mins was a confusion caused by the new way the builds are handled now. There was a retry of another build (the SonarCloud step), and that retry is what showed the increase.

I pushed a dummy commit, and this time the build passed with no retries and the time is the same as before.

So to answer your question: these tests do not add time to the JDK build. The service modules are already built at that point, and those already built modules are reused by the JUnit tests, which take a minute or less.
IMO we donot need a separate test build just for this .