feat(iceberg): thread per-file sort order into FileScanTask - #3128
feat(iceberg): thread per-file sort order into FileScanTask#3128anuragmantri wants to merge 3 commits into
Conversation
e285cc5 to
187999e
Compare
|
@mbutrovich - Since you have context on this work, could you please take a look? Thanks! |
anoopj
left a comment
anoopj
left a comment
There was a problem hiding this comment.
Code looks right to me. Just one comment about unsorted files.
| pub unified_partition_type: Option<Arc<StructType>>, | ||
|
|
||
| /// The sort order that this file's rows are sorted by, resolved from the data file's | ||
| /// `sort_order_id` against the table's known sort orders. `None` if the file has no |
There was a problem hiding this comment.
Note that Iceberg reserves 0 for unsorted. Per the spec:
Order id 0 is reserved for the unsorted order.
So a file with sort_order_id = 0 will resolve to unsorted order whereas a missing or unresolvable id gives None. So the datafusion consumer will need to check for both. e.g. something like sort_order.as_ref().is_some_and(|o| !o.is_unsorted())
May be a good idea to clarify this in the docs.
There was a problem hiding this comment.
Good catch. I changed it so that Some is returned only when the file has valid sort order (>0).
|
Thanks for the review @anoopj. I addressed your feedback. |
|
@CTTY @blackmwk @laskoviymishka Could you please review? |
2 participants
Which issue does this PR close?
What changes are included in this PR?
Resolves each manifest entry's
DataFile.sort_order_idagainst the table's known sort orders and carries the result on a newFileScanTask.sort_order field. This follows the same PlanContext -> ManifestFileContext -> ManifestEntryContext plumbing already used for unified_partition_type. None means no sort order could be established, either because the file has no recordedsort_order_id, or because the id doesn't resolve against the table's known sort orders.This is groundwork for propagating sort-order awareness into the DataFusion integration (Part 2 of #3126), which needs per-file sort-order data to decide whether a scan's output can be safely reported as sorted.
Are these changes tested?
Yes, a unit test covers all three resolution outcomes:
The manifest-writing setup for this is factored into a new shared fixture helper alongside the existing
setup_*variants.AI Disclosure
I'm new to Rust and this codebase. I used Claude code (Opus 5) to assist me with the PR and the test coverage and I manually reviewed it. Please bear with me as I get familiar with Rust and the codebase.