Skip to content

feat: druid.s3.enableLegacyMd5 property added - #20055

Open
nozjkoitop wants to merge 3 commits into
apache:masterfrom
deep-bi:feature-aws-sdk-legacy-mode-switch
Open

feat: druid.s3.enableLegacyMd5 property added#20055
nozjkoitop wants to merge 3 commits into
apache:masterfrom
deep-bi:feature-aws-sdk-legacy-mode-switch

Conversation

@nozjkoitop

Copy link
Copy Markdown
Contributor

Description

The AWS SDK v2 version used by Druid sends CRC32 request checksums by default for S3 operations such as DeleteObjects. Some S3-compatible storage implementations do not support these checksum headers and reject requests with a content-hash mismatch, causing operations such as bulk segment kill tasks to fail

This PR adds the druid.s3.enableLegacyMd5 configuration property. When enabled, Druid applies the SDK-provided LegacyMd5Plugin to both synchronous and asynchronous S3 clients, restoring Content-MD5 request checksums

The option is disabled by default, leaving the existing behavior unchanged for AWS S3 and compatible storage implementations that support CRC32 checksums

Release note

S3-compatible storage users can set druid.s3.enableLegacyMd5=true to use legacy MD5 request checksums. The default remains false


Key changed/added classes in this PR
  • AWSClientConfig
  • S3Utils
  • ServerSideEncryptingAmazonS3

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 1
P2 0
P3 0
Total 1

Reviewed 8 of 8 changed files.

Validation: git diff --check cb26014728ab3239bc8d7391f9cf6466e63905b0...HEAD passed. Builds and tests were not run.


This is an automated review by Codex GPT-5.6-Luna(max)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants