[SPARK-58965][SQL][PYTHON] Add gcd and lcm math functions - #58242
Open
Vivek1106-04 wants to merge 1 commit into
Open
[SPARK-58965][SQL][PYTHON] Add gcd and lcm math functions#58242Vivek1106-04 wants to merge 1 commit into
Vivek1106-04 wants to merge 1 commit into
Conversation
Spark SQL has no expression, and no combination of existing built-ins, that computes the greatest common divisor or the least common multiple of two integers, so users have to fall back to a UDF. This adds `gcd(expr1, expr2)` and `lcm(expr1, expr2)`. Both arguments are implicitly cast to BIGINT and the result is BIGINT, matching the existing `factorial` expression. The result is never negative, `gcd(0, 0)` and `lcm(0, 0)` are 0, and a NULL argument yields NULL. The greatest common divisor is computed with the Euclidean algorithm. The least common multiple divides by it before multiplying, so a result that is representable does not overflow on the way there. Results that are genuinely unrepresentable raise ARITHMETIC_OVERFLOW under ANSI mode and return NULL otherwise, as elsewhere in Spark. Both helpers live in MathUtils and are shared by the interpreted and codegen paths. PostgreSQL 13+ and DuckDB provide the same two functions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds two new built-in math functions,
gcdandlcm:Both arguments are implicitly cast to BIGINT and the result is BIGINT, matching the existing
factorialexpression, which likewise takes an integral argument and returns BIGINT.Semantics:
gcdlcmabs(other)ARITHMETIC_OVERFLOWunder ANSI mode, NULL otherwiseARITHMETIC_OVERFLOWunder ANSI mode, NULL otherwiseThe greatest common divisor is computed with the Euclidean algorithm. The least common multiple
divides by the greatest common divisor before multiplying, so a representable result never
overflows on the way there — for example
lcm(4611686018427387904, 2)returns4611686018427387904even though the naive product
4611686018427387904 * 2would not fit.Overflow arises in exactly two places, and in both PostgreSQL raises as well:
gcdwhere the result would be-Long.MinValue, which is not representable — that is, theinput pairs
(0, x),(x, 0)and(x, x)forx = Long.MinValue.lcmwhereabs(a) / gcd(a, b) * abs(b)exceedsLong.MaxValue.Both are reported with the existing
ARITHMETIC_OVERFLOWerror condition, so no new errorcondition is introduced. Overflow is gated on ANSI mode as it is elsewhere in Spark —
convin thesame file takes the same approach — raising under ANSI mode and returning NULL otherwise.
Both helpers live in
MathUtilsand are shared by the interpreted and codegen paths so the twocannot diverge.
The functions are exposed through SQL, the Scala/Java
functionsAPI, PySpark, and Spark Connect.Why are the changes needed?
Spark SQL currently has no way to compute either value. There is no expression for it, and no
combination of existing built-ins produces the result, so users have to fall back to a UDF — which
for PySpark means a Python round trip per row and no whole-stage codegen.
Both functions are standard in comparable engines:
gcd(a, b),lcm(a, b)forinteger,bigintandnumericgcd(a, b),lcm(a, b)(aliasesgreatest_common_divisor,least_common_multiple)Common uses include reducing fractions and ratios to lowest terms, aligning batch or partition
sizes, computing the repeat period of overlapping schedules, and normalizing denominators before
aggregation.
Does this PR introduce any user-facing change?
Yes. Two new built-in functions are available in SQL, the Scala/Java
functionsAPI, PySpark andSpark Connect. Previously
gcdandlcmwere unresolved function names:No existing behavior changes; the change is purely additive.
How was this patch tested?
New tests:
MathExpressionsSuite— unit tests for both expressions covering ordinary values, negativeinputs, zero, NULL, the boundary values around
Long.MinValue/Long.MaxValue, and overflow inboth ANSI and non-ANSI mode, plus interpreted-vs-codegen consistency checks.
math.sqlgolden-file tests, regenerated for the ANSI and non-ANSI results and the analyzerresults.
PlanGenerationTestSuitefunction tests, with the regenerated Spark Connect plan and explaingolden files.
gcdandlcm.Existing suites re-run and passing:
MathExpressionsSuite,SQLQueryTestSuite(math.sql),ExpressionsSchemaSuite,PlanGenerationTestSuite,ProtoToParsedPlanTestSuite.dev/lint-scalaanddev/lint-pythonboth pass.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)