Skip to content

[filesystem] Align Hadoop dependencies to 3.4.3#3699

Draft
litiliu wants to merge 6 commits into
apache:mainfrom
litiliu:codex/fluss-3682-hadoop-343
Draft

[filesystem] Align Hadoop dependencies to 3.4.3#3699
litiliu wants to merge 6 commits into
apache:mainfrom
litiliu:codex/fluss-3682-hadoop-343

Conversation

@litiliu

@litiliu litiliu commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3682

Fluss currently uses inconsistent Hadoop versions (fluss.hadoop.version=3.4.0, fs.hadoopshaded.version=3.3.4). This PR aligns all filesystem plugins to Hadoop 3.4.3 and migrates fluss-fs-s3 from the end-of-life AWS SDK for Java v1 to the modular AWS SDK v2, while preserving authentication, STS, KMS and S3 transfer-manager support.

Brief change log

  • Bump fluss.hadoop.version 3.4.0 → 3.4.3 and fs.hadoopshaded.version 3.3.4 → 3.4.3.
  • Migrate fluss-fs-s3 to AWS SDK v2 (software.amazon.awssdk): STS / KMS / auth / s3 / s3-transfer-manager as individual modules; exclude software.amazon.awssdk:bundle; drop the unused DynamoDB dependency.
  • Rewrite S3DelegationTokenProvider and DynamicTemporaryAWSCredentialsProvider against SDK v2 APIs; remove the copied XmlResponsesSaxParser and its checkstyle/coverage suppressions.
  • Exclude netty / dropwizard-metrics / jettison that are not used by hadoop-common; refresh NOTICE/LICENSE for all filesystem modules (including COS); add the reactive-streams license.

Artifact size review

Artifact Before After Change
Hadoop shaded 22.20 MB 24.13 MB +1.94 MB (+8.72%)
HDFS 34.66 MB 35.29 MB +0.63 MB (+1.82%)
S3 31.39 MB 44.52 MB +13.14 MB (+41.86%)
OSS 38.35 MB 28.40 MB −9.95 MB (−25.95%)
COS 41.63 MB 39.84 MB −1.79 MB (−4.30%)
GS 65.78 MB 67.71 MB +1.94 MB (+2.94%)
Azure 38.35 MB 30.80 MB −7.54 MB (−19.68%)
Total 272.35 MB 270.70 MB −1.65 MB (−0.61%)

The S3 increase is primarily the AWS SDK V2 S3 model and its asynchronous Netty transport. The s3-transfer-manager classes themselves account for only about 0.20 MB compressed, but Hadoop 3.4.3 S3A invokes S3TransferManager.copy() on copy/rename paths, so its module and transport cannot be removed safely. The resulting S3 artifact contains STS, KMS, Transfer Manager, and the async transport, and contains neither DynamoDB nor the complete AWS SDK bundle.

Tests

  • Updated S3DelegationTokenProviderTest for the SDK v2 credential APIs.
  • Existing filesystem IT cases (S3 / OSS / COS / GS / Azure / HDFS).
  • mvn clean verify, spotless and RAT.

API and Format

No public API or storage-format change. Note: s3.aws.credentials.provider (fs.s3a.aws.credentials.provider) now requires an AWS SDK v2 software.amazon.awssdk.auth.credentials.AwsCredentialsProvider implementation instead of an SDK v1 provider (documented).

Documentation

Updated the S3 and HDFS filesystem docs for AWS SDK v2 / Hadoop 3.4.3.

Generative AI disclosure

  • Yes (please specify the tool below)

Generated-by: OpenAI Codex following the guidelines (https://github.com/apache/fluss/blob/main/AGENTS.md)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[build/filesystem] Align Hadoop dependencies to 3.4.3

1 participant