feat: support case-insensitive column matching on reads (read-time parameter)#496
Open
JunRuiLee wants to merge 4 commits into
Open
feat: support case-insensitive column matching on reads (read-time parameter)#496JunRuiLee wants to merge 4 commits into
JunRuiLee wants to merge 4 commits into
Conversation
e671a08 to
9a2131c
Compare
df7d35c to
7ae85fc
Compare
Add ReadBuilder::with_case_sensitive(bool) (default true) and PredicateBuilder::new_with_case_sensitive(fields, bool), mirroring Java's RowType.getFieldIndex(name, caseSensitive). When disabled, projection and predicate column names resolve by ASCII case-folding, error on ambiguity, and the canonical schema name is stored in the predicate leaf. Projection resolution is deferred to scan/read build time so with_projection and with_case_sensitive are order-independent; with_projection does best-effort early validation, rejecting only columns that cannot match under any case sensitivity. Default stays case-sensitive.
Derive case sensitivity from the session (!enable_ident_normalization) for scan and filter translation, and carry it into the physical scan so execute() resolves column names the same way planning did. classify_filter_pushdown (no Session available) classifies case-sensitively and refuses Exact for ASCII-case-colliding schemas so a needed residual filter is never dropped.
…itive Thread the read-time flag into projection and dict_to_predicate; add the type stub and document that it must precede with_filter for predicates.
Add paimon_read_builder_with_case_sensitive and a case_sensitive parameter on the paimon_predicate_* constructors (threaded into integer coercion and PredicateBuilder). Projection names are validated for obvious typos early and resolved lazily. Covers Doris and the Go binding.
7ae85fc to
242ce95
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #495
Spark + Java Paimon support case-insensitive column matching on reads. In Java this lives at the engine / catalog layer — Spark's
spark.sql.caseSensitive(a session setting) drives Catalyst to resolve projection/filter columns case-insensitively and normalize them to the schema names before pushdown; Paimon core exposes it as a per-call parameter (RowType.getFieldIndex(name, caseSensitive)). It is not a table property.paimon-rusthad no equivalent — column resolution was always exact-case — so engines reading Paimon through the Rust core / C bindings / DataFusion / Python (e.g. Apache Doris) couldn't offer the behavior users get from Spark.This PR adds case-insensitive column matching as a read-time parameter (mirroring Java's per-call
caseSensitive), not a table option. Default stays case-sensitive, so existing behavior is unchanged; a reader opts in per read. When enabled, column names resolve by ASCII case-folding and an ambiguous reference (schema fields colliding under folding) is reported, mirroring Spark'sAMBIGUOUSbehavior.Brief change log
ReadBuilder::with_case_sensitive(bool)(defaulttrue) andPredicateBuilder::new_with_case_sensitive(fields, bool)(the existingnewdelegates withtrue). When disabled, column resolution folds case, errors on ambiguity, and stores the canonical schema name in the predicate leaf so downstream name-based lookups (e.g. index pruning) keep matching. Projection duplicate detection folds case so a case-colliding projection is reported as a duplicate.with_projectionandwith_case_sensitivemay be called in any order.with_projectiondoes best-effort early validation, rejecting only columns that cannot match under any case sensitivity (an obvious typo); case-dependent outcomes surface fromnew_read.!sql_parser.enable_ident_normalization) for scan and filter translation, and carries it into the physicalPaimonTableScansoexecute()resolves names the same way planning did.supports_filters_pushdownhas noSession, soclassify_filter_pushdownclassifies case-sensitively and refuses to reportExactfor a schema with ASCII-case-folding collisions — this guarantees a needed residual filter is never dropped even whenscanresolves case-insensitively.PyReadBuilder.with_case_sensitive(bool)threads the flag into projection and predicate conversion (type stub added).paimon_read_builder_with_case_sensitive(rb, bool)and acase_sensitiveparameter on thepaimon_predicate_*constructors (threaded into integer-datum coercion andPredicateBuilder). Covers Doris and the Go binding.ASCII-only folding throughout (
eq_ignore_ascii_case); no Unicode case folding.Scope: read path, top-level columns only. Nested/dotted paths, the
_ROW_IDsystem column (kept an exact reserved name), and the write path are out of scope.Tests
cargo test -p paimon --lib— projection/predicate case-insensitive match, default-sensitive rejection, ambiguity error, canonical-name storage, duplicate detection, order-independence (projection ↔ case-sensitive either order), and deferral of case-dependent errors tonew_read.cargo test -p paimon-datafusion --lib filter_pushdown/physical_plan::scan— session-derived pushdown, the classify collision guard, and partition planning.PYO3_PYTHON=python3.12).API and Format
ReadBuilder::with_case_sensitive,PredicateBuilder::new_with_case_sensitive; Cpaimon_read_builder_with_case_sensitiveand acase_sensitiveparameter added to thepaimon_predicate_*functions. ExistingPredicateBuilder::newis unchanged (case-sensitive).Documentation
Behavior described above and in the linked issue; the switch is a read-time API parameter, controlled by the reader/engine (not stored on the table), and is order-independent with projection.