Skip to content

Add list-aware physical scan split boundaries - #9578

Draft
mhk197 wants to merge 1 commit into
developfrom
mk/list-aware-split-boundaries-develop
Draft

Add list-aware physical scan split boundaries#9578
mhk197 wants to merge 1 commit into
developfrom
mk/list-aware-split-boundaries-develop

Conversation

@mhk197

@mhk197 mhk197 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

List layouts now persist exact scan split boundaries when an elements physical chunk end coincides with an enclosing list offset.

  • Add writer-side physical-boundary reports, scoped to each child writer invocation.
  • Repartition list elements only at enclosing-list offset boundaries, then make each chosen fence a physical ChunkedLayout seam before nested structural writers run.
  • Persist exact (outer_row_end, element_row_end) pairs in list metadata and use them directly during scan splitting; legacy layouts retain the existing heuristic fallback.
  • Configure the production file strategy to use the data-block target for list-aware element repartitioning and only coalesce whole selected chunks.
  • Propagate reports through chunked, dictionary, zoned, and structural writers; a struct reports only boundaries shared by all children.

Why

Previously, an elements layout could independently repartition in element-row space. Its physical endpoints almost never coincided with variable-length list input-chunk ends, leaving the parent list writer with few or no exact split points.

The parent list writer now owns a sparse, offset-aligned fence schedule. It does not retain a whole column of offsets: it translates only producer-selected candidates for each input chunk, and persists a candidate only after the completed child writer confirms that the fence stayed physical.

For nested list<list<...>> and list<struct<list<...>>>, each enclosing list materializes its own fences before invoking its structural elements writer. Consequently, inner writers cannot merge through an outer fence, and every metadata boundary remains in the appropriate enclosing list row space.

Test plan

  • cargo +nightly fmt --all
  • cargo test -p vortex-layout --no-fail-fast
  • cargo test -p vortex-file --no-fail-fast
  • cargo clippy -p vortex-layout -p vortex-file --all-targets --all-features -- -D warnings

Includes unit coverage for offset-aligned repartitioning, writer-report propagation, persisted-boundary reading, nested lists, and list-of-struct-of-list file splits.

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant