Skip to content

fix(parquet): coerce INT96 inside legacy Parquet list layouts - #26588

Open
DsThakurRawat wants to merge 2 commits into
apache:mainfrom
DsThakurRawat:fix/26393-coerce-int96-legacy-parquet-lists
Open

DsThakurRawat wants to merge 2 commits into
apache:mainfrom
DsThakurRawat:fix/26393-coerce-int96-legacy-parquet-lists

Conversation

@DsThakurRawat

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

In datafusion/datasource-parquet/src/schema_coercion.rs, Int96Coercer matches leaf column paths against int96_fields by traversing the schema with a DFS stack. When encountering a list field (DataType::List), the previous implementation hard-coded:

child_path.push(".list.");
child_path.push(unprocessed_child.name());

However, Parquet files written by Spark with legacy formatting (spark.sql.parquet.writeLegacyFormat=true), Hive, or older Parquet writers produce alternative list structures:

  • Spark legacy 3-level list: repeated group named bag, child element named array (e.g. at.bag.array).
  • Hive legacy 3-level list: repeated group named bag, child element named array_element (e.g. at.bag.array_element).
  • 2-level list: repeated primitive or group (e.g. at.array).
  • Repeated primitive at root or struct level without a synthetic list group.

Because child_path assumed .list.<child_name>, these legacy schemas resulted in non-matching paths such as at.list.array vs at.bag.array, leaving INT96 timestamps as Timestamp(Nanosecond, None) instead of coercing them to the target resolution (e.g., Timestamp(Microsecond, None)).

In addition, container types like DataType::LargeList and DataType::FixedSizeList were previously ignored by Int96Coercer.

What changes are included in this PR?

  • Pass Option<&'a ParquetType> in the StackContext tuple during schema DFS traversal to inspect the matching Parquet schema group/primitive types.
  • Introduce get_parquet_list_element_info:
    • For primitive repeated fields (2-level lists of primitives or repeated root fields), uses the actual field name without superfluous .list. segments.
    • For 3-level lists, dynamically extracts the repeated group name (list, bag, etc.) and element field name (element, array, array_element, etc.).
    • For 2-level lists of structs with multiple fields, resolves the repeated group directly as the struct element.
    • Gracefully falls back to standard conventions if Parquet type info is missing or unannotated.
  • Expand list pattern matching to support DataType::LargeList and DataType::FixedSizeList.
  • Check both int96_fields.contains(path) and current_parquet_type.is_some_and(|pt| pt.is_primitive() && pt.get_physical_type() == Type::INT96).

What is the testing strategy for this PR?

  • Added coerce_int96_to_resolution_with_legacy_list_layouts unit test covering:
    • standard 3-level (list/element)
    • spark_legacy 3-level (bag/array)
    • hive 3-level (bag/array_element)
    • two_level (array)
    • two_level_struct (array with multiple struct fields)
    • repeated_primitive_root (repeated int96 at)
    • nested_lists_legacy (multi-dimensional list of lists with legacy layouts)
  • Added coerce_int96_to_resolution_with_large_and_fixed_lists unit test testing LargeList and FixedSizeList containers.
  • Verified all 295 unit tests in datafusion-datasource-parquet pass without regressions.
  • Ran cargo fmt --check and cargo clippy -p datafusion-datasource-parquet --all-targets -- -D warnings.

Are there any user-facing changes?

No breaking changes. Queries reading Parquet files with legacy list structures containing INT96 timestamps will now correctly coerce to the configured timestamp resolution instead of retaining nanosecond precision.

Coerce INT96 physical timestamps inside legacy Parquet list layouts (Spark legacy bag/array, Hive legacy bag/array_element, two-level lists, and repeated primitive fields). Also add support for LargeList and FixedSizeList container types in Int96Coercer.

Closes apache#26393.

Signed-off-by: Divyansh Rawat <divyanshrawatofficial@gmail.com>
@github-actions github-actions Bot added the datasource Changes to the datasource crate label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

coerce_int96 does not coerce INT96 inside legacy Parquet list layouts (bag/array, two-level lists)

1 participant