Repository navigation
fix(parquet): coerce INT96 inside legacy Parquet list layouts - #26588
Open
DsThakurRawat wants to merge 2 commits into
Open
DsThakurRawat wants to merge 2 commits into
DsThakurRawat wants to merge 2 commits into
Conversation
Coerce INT96 physical timestamps inside legacy Parquet list layouts (Spark legacy bag/array, Hive legacy bag/array_element, two-level lists, and repeated primitive fields). Also add support for LargeList and FixedSizeList container types in Int96Coercer. Closes apache#26393. Signed-off-by: Divyansh Rawat <divyanshrawatofficial@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
In
datafusion/datasource-parquet/src/schema_coercion.rs,Int96Coercermatches leaf column paths againstint96_fieldsby traversing the schema with a DFS stack. When encountering a list field (DataType::List), the previous implementation hard-coded:However, Parquet files written by Spark with legacy formatting (
spark.sql.parquet.writeLegacyFormat=true), Hive, or older Parquet writers produce alternative list structures:bag, child element namedarray(e.g.at.bag.array).bag, child element namedarray_element(e.g.at.bag.array_element).at.array).Because
child_pathassumed.list.<child_name>, these legacy schemas resulted in non-matching paths such asat.list.arrayvsat.bag.array, leaving INT96 timestamps asTimestamp(Nanosecond, None)instead of coercing them to the target resolution (e.g.,Timestamp(Microsecond, None)).In addition, container types like
DataType::LargeListandDataType::FixedSizeListwere previously ignored byInt96Coercer.What changes are included in this PR?
Option<&'a ParquetType>in theStackContexttuple during schema DFS traversal to inspect the matching Parquet schema group/primitive types.get_parquet_list_element_info:.list.segments.list,bag, etc.) and element field name (element,array,array_element, etc.).DataType::LargeListandDataType::FixedSizeList.int96_fields.contains(path)andcurrent_parquet_type.is_some_and(|pt| pt.is_primitive() && pt.get_physical_type() == Type::INT96).What is the testing strategy for this PR?
coerce_int96_to_resolution_with_legacy_list_layoutsunit test covering:standard3-level (list/element)spark_legacy3-level (bag/array)hive3-level (bag/array_element)two_level(array)two_level_struct(arraywith multiple struct fields)repeated_primitive_root(repeated int96 at)nested_lists_legacy(multi-dimensional list of lists with legacy layouts)coerce_int96_to_resolution_with_large_and_fixed_listsunit test testingLargeListandFixedSizeListcontainers.datafusion-datasource-parquetpass without regressions.cargo fmt --checkandcargo clippy -p datafusion-datasource-parquet --all-targets -- -D warnings.Are there any user-facing changes?
No breaking changes. Queries reading Parquet files with legacy list structures containing INT96 timestamps will now correctly coerce to the configured timestamp resolution instead of retaining nanosecond precision.