Repository navigation
feat(substrait): add a hook for consuming relation inputs - #26603
Open
alexanderbianchi wants to merge 1 commit into
Open
alexanderbianchi wants to merge 1 commit into
alexanderbianchi wants to merge 1 commit into
Conversation
alexanderbianchi
force-pushed
the
codex/substrait-concurrent-consumer
branch
from
October 23, 2026 19:54
58e10b7 to
ccbcea5
Compare
alexanderbianchi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #26602. This extracts the extension point from a larger concurrent-consumer change; parallel default-consumer behavior is isolated in stacked draft #26680.
Rationale for this change
Custom Substrait consumers cannot overlap independent JOIN/UNION input consumption while reusing upstream operator assembly. Those helpers await each child directly. Downstream consumers currently need to override the operators and wrap already-consumed children just to control scheduling.
What changes are included in this PR?
Add
SubstraitConsumer::consume_rel_inputs(&[&Rel]) -> Result<Vec<LogicalPlan>>and call it from JOIN and UNION assembly. The default consumes inputs sequentially and stops on the first consumption error. Overrides must return one plan per input in input order; assembly checks the count.This PR adds no branch cloning, concurrency, caching, semaphore, or dependency. JOIN/UNION assembly still handles predicates, qualification, and distinctness. Invalid JOINs with missing children return an error instead of panicking.
What is the testing strategy for this PR?
Tests exercise the public consumer trait: sequential default consumption with a gated first input, error short-circuiting, custom-hook dispatch for JOIN/UNION ALL/UNION DISTINCT, input order, and rejection of incorrect result counts. Existing Substrait integration tests cover operator semantics.
Validation passed on base
6bb92bb3f:cargo fmt --all.cargo clippy --profile ci --all-targets --all-features -- -D warnings.The optional standalone lint suite could not start because system Python lacks PyYAML; formatting and Clippy were run directly.
Are there any user-facing changes?
Adds a provided trait method, so existing consumer implementations remain source-compatible and keep sequential input consumption. Custom consumers can override scheduling without replacing operator assembly.