Skip to content

S3 endpoint resolution no longer applied at ExecutionInterceptor.modifyHttpRequest() in AWS SDK for Java v2 2.47+ (verified on 2.49.5), so bucket cannot be extracted from the request URL until beforeTransmission. #7208

Description

@pradeepagrawal8184

Describe the bug

While upgrading AWS SDK for Java v2 from 2.46.21 to 2.49.5, middleware/tests that inspect S3 HTTP requests in ExecutionInterceptor.modifyHttpRequest() can no longer extract the bucket from the URL.

Likely cause: Starting in 2.47.0, S3 endpoint resolution moved from the interceptor phase to a later pipeline stage. At modifyHttpRequest, the URL is not yet resolved — host is the generic regional endpoint and the bucket is absent from both host and path.

Reference: aws/aws-sdk-java-v2#6955

Questions:

  1. Is moving endpoint resolution out of the interceptor phase in 2.47+ intentional and permanent for S3?
  2. What is the recommended way for interceptors to read the resolved bucket/endpoint before transmission — only beforeTransmission, or an ExecutionAttribute such as RESOLVED_ENDPOINT?
  3. Should modifyHttpRequest documentation note that S3 bucket/host mapping is not applied at that stage?

--

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

The build/tests should not break if 2.49.x is compatible with older version 2.46.x

Current Behavior

Observed behaviour (SDK 2.49.5)

Interceptor hook Host Path Bucket extractable?
modifyHttpRequest s3.us-west-2.amazonaws.com /path/to/object.txt No
beforeTransmission my-test-bucket.s3.us-west-2.amazonaws.com /path/to/object.txt Yes

On 2.46.21: modifyHttpRequest exposed virtual-hosted style (<bucket>.s3.<region>.amazonaws.com).


Reproduction Steps

Standalone reproduction

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
         http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>repro</groupId>
    <artifactId>s3-endpoint-resolution-repro</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.release>17</maven.compiler.release>
        <!-- Change to 2.46.21 to compare previous behaviour -->
        <aws.sdk.version>2.49.5</aws.sdk.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>${aws.sdk.version}</version>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.10.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.2.5</version>
            </plugin>
        </plugins>
    </build>
</project>

src/test/java/S3EndpointResolutionReproTest.java

import org.junit.jupiter.api.Test;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
import software.amazon.awssdk.core.interceptor.Context;
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
import software.amazon.awssdk.http.SdkHttpRequest;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;

import java.util.concurrent.atomic.AtomicReference;

import static org.junit.jupiter.api.Assertions.*;

class S3EndpointResolutionReproTest {
    private static final String BUCKET = "my-test-bucket";
    private static final String KEY    = "path/to/object.txt";

    @Test
    void modifyHttpRequest_endpointNotYetResolved() {
        AtomicReference<SdkHttpRequest> captured = new AtomicReference<>();

        S3Client client = S3Client.builder()
                .region(Region.US_WEST_2)
                .credentialsProvider(StaticCredentialsProvider.create(
                        AwsBasicCredentials.create("AKIAFAKEFAKEFAKE", "fakeSecretKey1234567890123456789012")))
                .overrideConfiguration(ClientOverrideConfiguration.builder()
                        .addExecutionInterceptor(new ExecutionInterceptor() {
                            @Override
                            public SdkHttpRequest modifyHttpRequest(
                                    Context.ModifyHttpRequest context,
                                    ExecutionAttributes attrs) {
                                captured.set(context.httpRequest());
                                throw new RuntimeException("stop");
                            }
                        }).build())
                .build();

        assertThrows(RuntimeException.class, () ->
                client.getObject(GetObjectRequest.builder().bucket(BUCKET).key(KEY).build()));

        SdkHttpRequest req = captured.get();
        System.out.println("modifyHttpRequest host: " + req.getUri().getHost());
        System.out.println("modifyHttpRequest path: " + req.getUri().getRawPath());

        assertEquals("s3.us-west-2.amazonaws.com", req.getUri().getHost());
        assertEquals("/path/to/object.txt", req.getUri().getRawPath());
    }

    @Test
    void beforeTransmission_endpointResolved() {
        AtomicReference<SdkHttpRequest> captured = new AtomicReference<>();

        S3Client client = S3Client.builder()
                .region(Region.US_WEST_2)
                .credentialsProvider(StaticCredentialsProvider.create(
                        AwsBasicCredentials.create("AKIAFAKEFAKEFAKE", "fakeSecretKey1234567890123456789012")))
                .overrideConfiguration(ClientOverrideConfiguration.builder()
                        .addExecutionInterceptor(new ExecutionInterceptor() {
                            @Override
                            public void beforeTransmission(
                                    Context.BeforeTransmission context,
                                    ExecutionAttributes attrs) {
                                captured.set(context.httpRequest());
                                throw new RuntimeException("stop");
                            }
                        }).build())
                .build();

        assertThrows(RuntimeException.class, () ->
                client.getObject(GetObjectRequest.builder().bucket(BUCKET).key(KEY).build()));

        SdkHttpRequest req = captured.get();
        System.out.println("beforeTransmission host: " + req.getUri().getHost());
        System.out.println("beforeTransmission path: " + req.getUri().getRawPath());

        assertEquals("my-test-bucket.s3.us-west-2.amazonaws.com", req.getUri().getHost());
    }
}

Run

mvn test

Expected stdout on 2.49.5:

modifyHttpRequest host: s3.us-west-2.amazonaws.com
modifyHttpRequest path: /path/to/object.txt
beforeTransmission host: my-test-bucket.s3.us-west-2.amazonaws.com
beforeTransmission path: /path/to/object.txt

Set <aws.sdk.version>2.46.21</aws.sdk.version> to compare — on the older version, modifyHttpRequest should include the bucket in the hostname. No live AWS credentials or network access required.

Possible Solution

No response

Additional Information/Context

Environment

Item Value
SDK artifact software.amazon.awssdk:bundle:2.49.5 (also tested with modular software.amazon.awssdk:s3:2.49.5)
Previous working version 2.46.21
Java OpenJDK 17
S3 client Default S3Client.builder().region(Region.US_WEST_2) — no endpointOverride, no forcePathStyle
Operation GetObject(bucket="my-test-bucket", key="path/to/object.txt")

Dependency version verification

We use the bundle fat JAR (all SDK modules at one version). No mixed module versions.

mvn dependency:tree -Dincludes=software.amazon.awssdk

Output:

[INFO] +- software.amazon.awssdk:bundle:jar:2.49.5:compile

Please note that I am not looking for a fix for the S3 endpoint resolution behavior at this time. I was attempting to upgrade from 2.46.21 to 2.49.x to address Netty CVEs flagged in #7189, but the upgrade to 2.49.5 introduced regressions in our S3 request parsing (endpoint resolution timing in modifyHttpRequest). Resolving those requires significant changes across multiple components and thorough regression testing, so we are unable to adopt 2.49.x at this time.

AWS Java SDK version used

2.49.5

JDK version used

17

Operating System and version

Apple M5 Pro : 26.5.2

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugThis issue is a bug.needs-triageThis issue or PR still needs to be triaged.potential-regressionMarking this issue as a potential regression to be checked by team member

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions