fix: honor target sorted flag and preserve map field metadata in cast_map_to_map - #5227
Open
Smallfu666 wants to merge 1 commit into
Open
fix: honor target sorted flag and preserve map field metadata in cast_map_to_map#5227Smallfu666 wants to merge 1 commit into
Smallfu666 wants to merge 1 commit into
Conversation
…_map_to_map (apache#5097) - Rename-only fast path (unchanged key/value types + unchanged sort order) delegates to arrow's cast, which relabels to the target entries/key/value fields and preserves their metadata with no value transformation (the common Parquet "key_value" -> Spark "entries" case). - When a child type changes, recurse with Comet's Spark-compatible cast_array for that child; when the sort order downgrades (true -> false, which arrow refuses) hand-build the result. - Honor the target sorted flag (previously the source flag was used); reject the unrepresentable unsorted -> sorted case with an error (a defensive guard; not planner-reachable). - Result data_type() equals the requested target type (field names, metadata, nullability). Scope note: a valid map's entries STRUCT cannot carry a non-None null buffer through safe Arrow APIs (all-valid buffers normalize to None), so entries-level null-buffer preservation is not an observable property and is not claimed. Map-LEVEL null preservation is covered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to #5097. (Intentionally not
Closes— see the scope note; the "entries null buffer" part of the issue is not representable through safe Arrow APIs.)Rationale for this change
cast_map_to_maphad two real bugs and one latent hazard:sortedflag for the result, so a cast to a map type with a differentsortedflag returned the wrong type.data_type()did not equal the requested target.sortedmap can reorder keys (e.g. sorted Utf8["10","2"]→ Int32[10,2]), which would silently produce aMap(.., sorted=true)that is no longer sorted.What changes are included in this PR?
key_value→ Sparkentriesrelabel), delegate to arrow'scast, which relabels to the target fields and preserves their metadata with no value transformation.cast_array; when the sort order downgrades (true → false), hand-build with the target flag.try_newis used so a malformed target returnsErrinstead of panicking, and the resultdata_type()equals the requested target.to_sorted && (!from_sorted || key-type changed)— a cast cannot manufacture or preserve an ordering across a key-type change.[0]/[1]— a malformed entries struct (0/1/3+ fields) returnsErr, never panics.Scope note: an entries struct carrying a non-
Nonenull buffer is not constructible through the inspected safe Arrow constructors (StructArray::new/try_newandMapArraynormalize an all-valid buffer toNone), so entries-level null-buffer "preservation" is not an observable property and is not claimed. Map-level null preservation is covered.How are these changes tested?
16 unit tests in
cast.rs: rename-only fast path (full target schema + unchanged values/offsets/map nulls); sorted equal /true→false/false→true(Err) /true→truekey-type-change(Err) / value-only-cast; metadata + child casts; empty maps; mixed null/empty rows; sliced maps; key+value casts; malformed targets (0/1/3 entry fields, non-nullable field with null) returningErrwithout panic.