perf: skip DecimalPrecision.promote's rewrite when there is no decimal arithmetic - #5216
Open
andygrove wants to merge 2 commits into
Open
perf: skip DecimalPrecision.promote's rewrite when there is no decimal arithmetic#5216andygrove wants to merge 2 commits into
andygrove wants to merge 2 commits into
Conversation
…l arithmetic QueryPlanSerde.exprToProto runs DecimalPrecision.promote over every expression it converts. The rule is a transformUp, so it walks and rebuilds every node even when nothing matches, which is the common case. Guard it with a cheap traversal that looks for arithmetic producing a decimal, and only run the rewrite when one exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #5199 (item 4:
DecimalPrecision.promoteis a second full traversal of every expression tree).Rationale for this change
QueryPlanSerde.exprToProtocallsDecimalPrecision.promoteon every expression it converts, andpromoteis aTreeNode.transformUp.transformUpwalks the whole tree and callsmapChildrenatevery node, which allocates through
mapProductIteratorregardless of whether any case matches. Theresult is a full extra traversal of every expression tree before the serde walks it again, paid on
the driver for every query and again for every query stage under AQE.
The rule only ever rewrites arithmetic that produces a decimal, so on the overwhelming majority of
expressions the traversal produces the identical tree back.
Measured with a throwaway probe over decimal-free expression trees of varying size (Spark 4.1 /
JDK 17, 20k iterations per measurement, best of 5 after warm-up):
exprToProtobeforeTrees that do contain decimal arithmetic pay the guard traversal on top of the rewrite, but that
traversal short-circuits at the first match and does not allocate.
What changes are included in this PR?
Adds a
containsDecimalArithmeticguard and only runs thetransformUpwhen it holds.The guard is deliberately a superset of the rule: it matches
Add/Subtract/Multiply/Divide/RemainderwhosedataTypeis aDecimalType, ignoring the operands that the rule'sown patterns check. It is sound to decide this up front because the rewrite only wraps nodes in
CheckOverflow, andCheckOverflow's data type is its child's, so no node's data type changespart way through the
transformUpand a tree without decimal arithmetic cannot grow any.Behavior is unchanged:
transformUpalready returns the input tree by identity when no casematches, so the guard removes work rather than changing the result.
How are these changes tested?
Existing coverage:
CometExpressionSuite(140 tests) and the Spark 4.1CometDecimalArithmeticViewSuiteregression tests for #4124 and #5075 all pass.Added
DecimalPrecisionSuite, which pins the rule's behavior directly and is version-agnostic(the existing unit coverage was Spark 4.1 only). It asserts that trees without decimal arithmetic
come back by identity, that each of the five arithmetic operators is wrapped in
CheckOverflowwith the operator's own
dataTypeas the target, that a decimal operator buried under nodes therule does not rewrite is still promoted, and that a mixed tree has only its decimal arithmetic
rewritten. These pass both with and without this change, which is the point: they document that
the guard is not observable.
The probe used for the numbers above was throwaway and is not included.