[filesystem] Align Hadoop dependencies to 3.4.3#3699
Draft
litiliu wants to merge 6 commits into
Draft
Conversation
Migrate the S3 filesystem to modular AWS SDK v2 dependencies while preserving authentication, STS, KMS, and transfer-manager support.
2 tasks
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #3682
Fluss currently uses inconsistent Hadoop versions (
fluss.hadoop.version=3.4.0,fs.hadoopshaded.version=3.3.4). This PR aligns all filesystem plugins to Hadoop 3.4.3 and migratesfluss-fs-s3from the end-of-life AWS SDK for Java v1 to the modular AWS SDK v2, while preserving authentication, STS, KMS and S3 transfer-manager support.Brief change log
fluss.hadoop.version3.4.0 → 3.4.3 andfs.hadoopshaded.version3.3.4 → 3.4.3.fluss-fs-s3to AWS SDK v2 (software.amazon.awssdk): STS / KMS / auth / s3 / s3-transfer-manager as individual modules; excludesoftware.amazon.awssdk:bundle; drop the unused DynamoDB dependency.S3DelegationTokenProviderandDynamicTemporaryAWSCredentialsProvideragainst SDK v2 APIs; remove the copiedXmlResponsesSaxParserand its checkstyle/coverage suppressions.hadoop-common; refreshNOTICE/LICENSEfor all filesystem modules (including COS); add the reactive-streams license.Artifact size review
The S3 increase is primarily the AWS SDK V2 S3 model and its asynchronous Netty transport. The
s3-transfer-managerclasses themselves account for only about 0.20 MB compressed, but Hadoop 3.4.3 S3A invokesS3TransferManager.copy()on copy/rename paths, so its module and transport cannot be removed safely. The resulting S3 artifact contains STS, KMS, Transfer Manager, and the async transport, and contains neither DynamoDB nor the complete AWS SDK bundle.Tests
S3DelegationTokenProviderTestfor the SDK v2 credential APIs.mvn clean verify, spotless and RAT.API and Format
No public API or storage-format change. Note:
s3.aws.credentials.provider(fs.s3a.aws.credentials.provider) now requires an AWS SDK v2software.amazon.awssdk.auth.credentials.AwsCredentialsProviderimplementation instead of an SDK v1 provider (documented).Documentation
Updated the S3 and HDFS filesystem docs for AWS SDK v2 / Hadoop 3.4.3.
Generative AI disclosure
Generated-by: OpenAI Codex following the guidelines (https://github.com/apache/fluss/blob/main/AGENTS.md)