Skip to content

feat: support case-insensitive column matching on reads (read-time parameter)#496

Open
JunRuiLee wants to merge 4 commits into
apache:mainfrom
JunRuiLee:feat/case-insensitive-upstream
Open

feat: support case-insensitive column matching on reads (read-time parameter)#496
JunRuiLee wants to merge 4 commits into
apache:mainfrom
JunRuiLee:feat/case-insensitive-upstream

Conversation

@JunRuiLee

@JunRuiLee JunRuiLee commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #495

Spark + Java Paimon support case-insensitive column matching on reads. In Java this lives at the engine / catalog layer — Spark's spark.sql.caseSensitive (a session setting) drives Catalyst to resolve projection/filter columns case-insensitively and normalize them to the schema names before pushdown; Paimon core exposes it as a per-call parameter (RowType.getFieldIndex(name, caseSensitive)). It is not a table property.

paimon-rust had no equivalent — column resolution was always exact-case — so engines reading Paimon through the Rust core / C bindings / DataFusion / Python (e.g. Apache Doris) couldn't offer the behavior users get from Spark.

This PR adds case-insensitive column matching as a read-time parameter (mirroring Java's per-call caseSensitive), not a table option. Default stays case-sensitive, so existing behavior is unchanged; a reader opts in per read. When enabled, column names resolve by ASCII case-folding and an ambiguous reference (schema fields colliding under folding) is reported, mirroring Spark's AMBIGUOUS behavior.

Brief change log

  • Core: ReadBuilder::with_case_sensitive(bool) (default true) and PredicateBuilder::new_with_case_sensitive(fields, bool) (the existing new delegates with true). When disabled, column resolution folds case, errors on ambiguity, and stores the canonical schema name in the predicate leaf so downstream name-based lookups (e.g. index pruning) keep matching. Projection duplicate detection folds case so a case-colliding projection is reported as a duplicate.
  • Order-independent projection: projection name resolution is deferred to scan/read build time and uses the case sensitivity effective then, so with_projection and with_case_sensitive may be called in any order. with_projection does best-effort early validation, rejecting only columns that cannot match under any case sensitivity (an obvious typo); case-dependent outcomes surface from new_read.
  • DataFusion: derives case sensitivity from the session (!sql_parser.enable_ident_normalization) for scan and filter translation, and carries it into the physical PaimonTableScan so execute() resolves names the same way planning did. supports_filters_pushdown has no Session, so classify_filter_pushdown classifies case-sensitively and refuses to report Exact for a schema with ASCII-case-folding collisions — this guarantees a needed residual filter is never dropped even when scan resolves case-insensitively.
  • Python: PyReadBuilder.with_case_sensitive(bool) threads the flag into projection and predicate conversion (type stub added).
  • C binding: paimon_read_builder_with_case_sensitive(rb, bool) and a case_sensitive parameter on the paimon_predicate_* constructors (threaded into integer-datum coercion and PredicateBuilder). Covers Doris and the Go binding.

ASCII-only folding throughout (eq_ignore_ascii_case); no Unicode case folding.

Scope: read path, top-level columns only. Nested/dotted paths, the _ROW_ID system column (kept an exact reserved name), and the write path are out of scope.

Tests

  • cargo test -p paimon --lib — projection/predicate case-insensitive match, default-sensitive rejection, ambiguity error, canonical-name storage, duplicate detection, order-independence (projection ↔ case-sensitive either order), and deferral of case-dependent errors to new_read.
  • cargo test -p paimon-datafusion --lib filter_pushdown / physical_plan::scan — session-derived pushdown, the classify collision guard, and partition planning.
  • Python predicate/read tests (pyo3, PYO3_PYTHON=python3.12).

API and Format

  • New public API: ReadBuilder::with_case_sensitive, PredicateBuilder::new_with_case_sensitive; C paimon_read_builder_with_case_sensitive and a case_sensitive parameter added to the paimon_predicate_* functions. Existing PredicateBuilder::new is unchanged (case-sensitive).
  • No table option and no storage-format change. Default behavior is unchanged (case-sensitive).

Documentation

Behavior described above and in the linked issue; the switch is a read-time API parameter, controlled by the reader/engine (not stored on the table), and is order-independent with projection.

@JunRuiLee JunRuiLee marked this pull request as draft July 10, 2026 07:38
@JunRuiLee JunRuiLee force-pushed the feat/case-insensitive-upstream branch from e671a08 to 9a2131c Compare July 10, 2026 08:44
@JunRuiLee JunRuiLee changed the title feat: support case-insensitive column matching on reads feat: support case-insensitive column matching on reads (read-time option) Jul 10, 2026
@JunRuiLee JunRuiLee force-pushed the feat/case-insensitive-upstream branch 2 times, most recently from df7d35c to 7ae85fc Compare July 10, 2026 10:00
@JunRuiLee JunRuiLee marked this pull request as ready for review July 10, 2026 10:03
@JunRuiLee JunRuiLee changed the title feat: support case-insensitive column matching on reads (read-time option) feat: support case-insensitive column matching on reads (read-time parameter) Jul 10, 2026
@JunRuiLee JunRuiLee marked this pull request as draft July 10, 2026 11:24
Add ReadBuilder::with_case_sensitive(bool) (default true) and
PredicateBuilder::new_with_case_sensitive(fields, bool), mirroring Java's
RowType.getFieldIndex(name, caseSensitive). When disabled, projection and
predicate column names resolve by ASCII case-folding, error on ambiguity,
and the canonical schema name is stored in the predicate leaf.

Projection resolution is deferred to scan/read build time so
with_projection and with_case_sensitive are order-independent; with_projection
does best-effort early validation, rejecting only columns that cannot match
under any case sensitivity. Default stays case-sensitive.
Derive case sensitivity from the session (!enable_ident_normalization) for
scan and filter translation, and carry it into the physical scan so execute()
resolves column names the same way planning did. classify_filter_pushdown
(no Session available) classifies case-sensitively and refuses Exact for
ASCII-case-colliding schemas so a needed residual filter is never dropped.
…itive

Thread the read-time flag into projection and dict_to_predicate; add the
type stub and document that it must precede with_filter for predicates.
Add paimon_read_builder_with_case_sensitive and a case_sensitive parameter
on the paimon_predicate_* constructors (threaded into integer coercion and
PredicateBuilder). Projection names are validated for obvious typos early and
resolved lazily. Covers Doris and the Go binding.
@JunRuiLee JunRuiLee force-pushed the feat/case-insensitive-upstream branch from 7ae85fc to 242ce95 Compare July 10, 2026 12:07
@JunRuiLee JunRuiLee marked this pull request as ready for review July 10, 2026 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support case-insensitive column matching on reads (read-time parameter)

1 participant