Skip to content

feat: physical optimizer rules declare whether they run before or after enforcement #26979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jayzhan211 wants to merge 3 commits into
base: main
Choose a base branch
from
+670 −78
Open

feat: physical optimizer rules declare whether they run before or after enforcement #26979

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5b79e98
feat: physical optimizer rules declare whether they run before or aft…
jayzhan211 Nov 1, 2026
34301a6
docs: describe physical optimizer rule phases and who is affected
jayzhan211 Nov 1, 2026
a86d79d
feat: separate the enforcement phase so analyzer rules cannot run aft…
jayzhan211 Nov 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 32 additions & 24 deletions datafusion/core/src/optimizer_rule_reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,33 +67,41 @@ Rule order matters. The default pipeline may change between releases.
### Physical Optimizer Rules

The same rule name may appear more than once when the default pipeline runs it
in multiple phases.
in multiple passes.

Rules 1 to 7 are analyzer rules (`PhysicalOptimizerRule::phase` returns
`PhysicalOptimizerPhase::Analyzer`): they run before the plan meets its
distribution and ordering requirements. Rule 8, `EnsureRequirements`, is the
enforcement rule (`PhysicalOptimizerPhase::Enforcement`): it makes the
requirements hold. The other rules are optimizer rules: each receives a plan
that meets the requirements and must return one. The physical planner fails
planning when a rule list puts a rule before a rule of an earlier phase.

After the last rule, including any rule added by the user, the physical planner
checks that the final plan meets its ordering, distribution, and infinite-input
safety requirements (`SanityCheckPlan`). If the ordering or distribution
requirements do not hold, the error names the rule that broke them.

| order | rule | phase | summary |
| ----- | ------------------------------ | ----------------------- | -------------------------------------------------------------------------------------------------------- |
| 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. |
| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |
| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |
| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |
| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |
| 6 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |
| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |
| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |
| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |
| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |
| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |
| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |
| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |
| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |
| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |
| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |
| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |
| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |
| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |
| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |
| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |
| order | rule | pass | summary |
| ----- | ------------------------------ | ----------------- | -------------------------------------------------------------------------------------------------------- |
| 1 | `OutputRequirements` | add | Adds helper nodes so output requirements survive later physical rewrites. |
| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |
| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |
| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |
| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |
| 6 | `FilterPushdown` | pre-optimization | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |
| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |
| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |
| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |
| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |
| 11 | `ProjectionPushdown` | early | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |
| 12 | `OutputRequirements` | remove | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |
| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |
| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |
| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |
| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |
| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |
| 18 | `ProjectionPushdown` | late | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |
| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |
| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |
| 21 | `FilterPushdown(Post)` | post-optimization | Pushes dynamic filters at the end of optimization, after plan references stop moving. |
Loading
Loading