[AURON #2432] Vendor third-party dependencies via setup-vendors.sh with patch series - #2433
Open
diliulou wants to merge 1 commit into
Open
[AURON #2432] Vendor third-party dependencies via setup-vendors.sh with patch series#2433diliulou wants to merge 1 commit into
diliulou wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR replaces direct forked git-based Rust dependencies (Arrow/DataFusion/ORC/serde_json) with a vendored-sources workflow driven by a repository script plus an in-repo patch series, aligning dependency sourcing with ASF constraints.
Changes:
- Add
dev/vendors/setup-vendors.shto clone upstream repos at pinned refs and apply local patch series underdev/vendors/patches/**. - Switch Rust dependency overrides in
Cargo.tomlto local path-based[patch.crates-io]entries targetingdev/vendors/packages/**, and exclude vendor workspaces from the main workspace. - Integrate vendor setup into the native build flow and housekeeping via
.gitignore/.rat-excludes.
Reviewed changes
Copilot reviewed 30 out of 36 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| spark-extension/src/main/scala/org/apache/spark/sql/auron/util/TaskContextHelper.scala | Formatting-only adjustment to multiline string interpolation indentation. |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeHelper.scala | Formatting-only adjustment to log string indentation. |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronSparkSessionExtension.scala | Formatting-only adjustment to log string indentation. |
| spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala | Formatting-only adjustment to assertion/error string indentation. |
| dev/vendors/setup-vendors.sh | New script to clone upstream vendor repos and apply ordered patch series. |
| dev/vendors/patches/serde_json/0002-blaze-do-not-force-escaping-control-characters.patch | serde_json patch: adjust control-character escaping behavior. |
| dev/vendors/patches/serde_json/0001-blaze-ignore-invalid-escape.patch | serde_json patch: tolerate/handle invalid escape sequences. |
| dev/vendors/patches/orc-rust/0001-blaze-adapt-arrow-dependency-to-v55.2.0-blaze.patch | orc-rust patch: pin Arrow dependency versions to match vendored Arrow. |
| dev/vendors/patches/datafusion/0015-blaze-revert-substr-to-v45.0.0-for-unsupported-string-view.patch | datafusion patch: change substr typing/return behavior and tests. |
| dev/vendors/patches/datafusion/0014-blaze-implement-TestMemoryExec-as-any.patch | datafusion patch: implement as_any() for TestMemoryExec. |
| dev/vendors/patches/datafusion/0013-blaze-make-parquet-reader-components-public.patch | datafusion patch: widen visibility for parquet reader components/mapping. |
| dev/vendors/patches/datafusion/0012-blaze-supports-short-circuiting-And-Or.patch | datafusion patch: add short-circuiting AND/OR physical expressions and integrate in pruning. |
| dev/vendors/patches/datafusion/0011-blaze-supports-pruning-with-starts_with.patch | datafusion patch: add pruning rewrite support for starts_with. |
| dev/vendors/patches/datafusion/0010-blaze-use-case-insensitive-comparing-in-processing-parquet-f.patch | datafusion patch: use case-insensitive comparisons for schema matching. |
| dev/vendors/patches/datafusion/0009-blaze-disable-isnull-isnotnull-pruning-for-nested-data-type.patch | datafusion patch: skip null-pruning for nested data types. |
| dev/vendors/patches/datafusion/0008-blaze-use-deprecated-min-max-values-only-when-min-==-max.patch | datafusion patch: conditional use of deprecated min/max statistics. |
| dev/vendors/patches/datafusion/0007-blaze-make-scatter-public.patch | datafusion patch: re-export scatter helper publicly. |
| dev/vendors/patches/datafusion/0006-blaze-allow-nullable-parquet-scan-partition-key.patch | datafusion patch: allow nullable partition-key fields in parquet scans. |
| dev/vendors/patches/datafusion/0005-blaze-supports-parquet-dictionary-filtering.patch | datafusion patch: add dictionary-driven pruning support and related plumbing. |
| dev/vendors/patches/datafusion/0004-blaze-handle-special-cases-when-pruning-float-column-with-Na.patch | datafusion patch: handle NaN / signed-zero edge cases in pruning stats. |
| dev/vendors/patches/datafusion/0003-blaze-supports-parquet-scan-with-adaptive-batch-size.patch | datafusion patch: add adaptive parquet scan batch sizing. |
| dev/vendors/patches/datafusion/0002-blaze-exclude-testing-submodules.patch | datafusion patch: remove/disable testing submodules. |
| dev/vendors/patches/datafusion/0001-blaze-update-arrow-dependency.patch | datafusion patch: add path-based Arrow overrides for vendored Arrow workspace. |
| dev/vendors/patches/arrow-rs/0008-blaze-parquet-supports-processing-old-style-array-struct.patch | arrow-rs patch: extend schema handling for legacy array-of-struct representations. |
| dev/vendors/patches/arrow-rs/0007-blaze-use-bufreader-to-read-page-header-avoid-reading-in-fragments.patch | arrow-rs patch: BufReader optimization and sync bytes range type adjustment. |
| dev/vendors/patches/arrow-rs/0006-blaze-expose-fields-of-AsyncReader-ArrowReaderBuilder-to-pub.patch | arrow-rs patch: widen visibility of ArrowReaderBuilder/AsyncReader internals. |
| dev/vendors/patches/arrow-rs/0005-blaze-fix-statistics-null-count-is-negative-error.patch | arrow-rs patch: clamp negative null-count stats. |
| dev/vendors/patches/arrow-rs/0004-blaze-map-field-name-fixed-to-entries.patch | arrow-rs patch: fix map field name to entries and add sync bytes hook. |
| dev/vendors/patches/arrow-rs/0003-blaze-skip-parquet-utf-8-validating.patch | arrow-rs patch: disable UTF-8 validation paths for parquet/arrow string data. |
| dev/vendors/patches/arrow-rs/0002-blaze-add-get_dictionary.patch | arrow-rs patch: add dictionary extraction helper for pruning and expose needed modules. |
| dev/vendors/patches/arrow-rs/0001-blaze-fix-NaN-comparison-semantic.patch | arrow-rs patch: adjust NaN ordering/equality semantics. |
| dev/mvn-build-helper/build-native.sh | Run vendor setup before native build to ensure path-patched crates exist. |
| Cargo.toml | Exclude vendor workspaces and switch [patch.crates-io] overrides to local vendored paths. |
| Cargo.lock | Refresh lockfile after moving from git sources to local patched paths (and other resolution changes). |
| .rat-excludes | Exclude vendor-related paths from RAT scanning (as currently configured). |
| .gitignore | Ignore generated dev/vendors/packages/ checkouts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
diliulou
changed the title
richox
force-pushed
the
dev-setup-vendors
branch
from
July 27, 2026 04:30
8fd5a5e to
78d7cab
Compare
richox
force-pushed
the
dev-setup-vendors
branch
from
July 27, 2026 04:30
78d7cab to
c4be2a8
Compare
richox
force-pushed
the
dev-setup-vendors
branch
from
July 27, 2026 06:41
c4be2a8 to
fcc19e6
Compare
Replace direct git/rev dependencies for datafusion, arrow-rs, orc-rust,
and serde_json with a vendor+patch approach:
- Add dev/vendors/setup-vendors.sh to clone upstream repos at pinned tags
and apply local patch series
- Add patch files under dev/vendors/patches/{datafusion,arrow-rs,orc-rust,serde_json}/
- Update Cargo.toml: use relative path patches to vendored packages,
exclude vendor workspaces from the main workspace
- Update .gitignore to ignore dev/vendors/packages/
- Update .rat-excludes for dev/vendors/**
- Call setup-vendors.sh all from build-native.sh before cargo build
richox
force-pushed
the
dev-setup-vendors
branch
from
July 27, 2026 06:54
fcc19e6 to
b237fc6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace direct git/rev dependencies for datafusion, arrow-rs, orc-rust, and serde_json with a vendor+patch approach:
Which issue does this PR close?
Closes #2432
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?
No.
How was this patch tested?
Via github CI.
Was this patch authored or co-authored using generative AI tooling?
If yes, include:
Generated-by: GLM-5.2ASF guidance: https://www.apache.org/legal/generative-tooling.html