Skip to content

fix: UNNEST now preserves item fields in expression inference and logical-plan schemas - #26433

Merged
alamb merged 7 commits into
apache:mainfrom
quwin:fix/unnest-drops-metadata
Oct 20, 2026
Merged

alamb merged 7 commits into
apache:mainfrom
quwin:fix/unnest-drops-metadata

Conversation

@quwin

@quwin quwin commented Oct 16, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

In SedonaDB, unnesting a list of GeoArrow geometries currently returns plain binary values, losing the geometry type and CRS metadata needed by downstream spatial operations; metadata-aware SQL functions can also reject the field during planning.

What changes are included in this PR?

  • Preserve list-item fields in expression inference and logical-plan schemas while retaining output naming and nullability behavior.

What is the testing strategy for this PR?

  • DataFrame regressions cover all five list variants, nested unnesting, aliases, NULL/empty lists, and metadata-sensitive UDF planning. The targeted regression run passed all 7 cases.
  • The logical_unnest_plus_1000_columns benchmark in the optimized release-nonlto profile measured 47.58 ms on base 0d79252f and 48.75 ms on this branch (+2.5% median). This is within the noise threshold.
  • The full extended workspace suite and all-features Clippy were not rerun; the existing Clippy issue Clippy fails with all features enabled #26181 remains noted.

Are there any user-facing changes?

UNNEST retains list-item metadata, including Arrow extension metadata, in inferred fields and result schemas. No public Rust API changes.

@github-actions github-actions Bot added logical-expr Logical plan and expressions core Core DataFusion crate labels Oct 16, 2026
@quwin
quwin marked this pull request as ready for review October 17, 2026 03:18
@quwin

quwin commented Oct 17, 2026

Copy link
Copy Markdown
Author

@jayzhan211 Could you review this UNNEST bug fix and run CI? Thanks!

@jayzhan211
jayzhan211 self-requested a review October 19, 2026 12:03
Comment thread datafusion/expr/src/expr_schema.rs

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @quwin , overall LGTM

@alamb
alamb enabled auto-merge October 19, 2026 20:13
@github-actions github-actions Bot added auto detected api change Auto detected API change and removed auto detected api change Auto detected API change labels Oct 20, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.36585% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.93%. Comparing base (d89a85d) to head (049fbd2).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/expr/src/expr_schema.rs 83.33% 0 Missing and 3 partials ⚠️
datafusion/expr/src/utils.rs 88.23% 2 Missing ⚠️
datafusion/expr/src/logical_plan/plan.rs 83.33% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26433      +/-   ##
==========================================
- Coverage   82.93%   82.93%   -0.01%     
==========================================
  Files        1154     1154              
  Lines      457933   457954      +21     
  Branches   457933   457954      +21     
==========================================
+ Hits       379786   379796      +10     
- Misses      55128    55133       +5     
- Partials    23019    23025       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alamb
alamb added this pull request to the merge queue Oct 20, 2026
Merged via the queue into apache:main with commit 79650ec Oct 20, 2026
42 checks passed
@quwin
quwin deleted the fix/unnest-drops-metadata branch October 20, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unnest drops the list item field's metadata (e.g. Arrow extension types)

5 participants