feat: support make_interval via codegen dispatch - #5260
Conversation
MakeInterval has no Comet handler today, so any query using make_interval falls the whole operator back to Spark. Route it through the JVM codegen dispatcher, which runs Spark's own generated code inside the native Comet pipeline and so matches Spark bit-for-bit across all supported versions. make_interval produces CalendarIntervalType, which Comet's columnar layer gained support for in apache#4898; that is what makes carrying the dispatcher's output possible now. Adds make_interval.sql and make_interval_ansi.sql fixtures, the latter covering the ANSI overflow path where MakeInterval.failOnError is true and overflow raises ARITHMETIC_OVERFLOW instead of returning NULL.
Just to clarify — is this planned for 1.0.0 so we can deliver the native implementation without schedule pressure? |
There was a problem hiding this comment.
we could also add test cases which is from the review of native make_interval impl: #5039 (comment)
Spark's own sql-tests/inputs/interval.sql exercises exactly this range:
select make_interval(1, 2, 3, 4, 0, 0, 123456789012.123456);
There was a problem hiding this comment.
other reference: #5039 (comment)
Spark's own IntervalExpressionsSuite / interval.sql exercise that aren't covered yet:
- Microsecond-precision seconds like Spark's docstring example make_interval(0, 1, 0, 1, 0, 0, 100.000001) asserted directly (it's currently only exercised via the column path where it can be hard to spot a per-row precision drift).
- Nulls in components other than years in the column path (currently only the years=NULL row is tested).
- Large-second cases from Spark's interval.sql: make_interval(1, 2, 3, 4, 0, 0, 123456789012.123456) and make_interval(0, 0, 0, 0, 0, 0, 1234567890123456789). If either is a known divergence (see the nanos-overflow comment on the Rust file), wrapping them in query ignore() would at least pin the behavior for future readers.
- Int.MinValue for a signed-overflow smoke test on the years column.
There was a problem hiding this comment.
ditto as `make_interval.sql.
ref: #5039 (comment)
Spark's IntervalExpressionsSuite ANSI mode block covers weeks = Int.MaxValue, and per-row overflow via hours/mins/seconds interactions. Something like:
query expect_error(overflow) SELECT make_interval(0, 0, 2147483647)
Hi @peterxcli This work was part of a PR that was created before your PR. We can review both and see what makes sense (maybe even hybrid). Let me know when your PR is ready for another review. |
Which issue does this PR close?
Closes #3099. Part of the interval support epic #5061.
Split out of #5030, which originally carried this alongside
timestampaddandtimestampdiff. Those two are unrelated tomake_intervalbeyond sharing the codegen-dispatch mechanism, so they now travel separately.Note there is an alternative native implementation in #5039, which wires
make_intervalto DataFusion'sSparkMakeInterval. The two approaches are mutually exclusive; see the trade-off below.Rationale for this change
MakeIntervalhas no Comet handler today, so any query usingmake_intervalfalls the entire operator back to Spark. It is a regular expression, notRuntimeReplaceable, so it reaches serde directly.This PR routes it through the JVM codegen dispatcher rather than adding a native implementation. The dispatcher runs Spark's own generated code inside the native Comet pipeline, which keeps the operator native while guaranteeing bit-for-bit Spark compatibility. For
make_intervalspecifically this covers three behaviors that are easy to diverge on natively:secondsargument is aDecimal(18, 6)scaled to microseconds, with its own overflow check;failOnError(defaulting toSQLConf.ansiEnabled) chooses between raisingARITHMETIC_OVERFLOWand returning NULL, and it must be the same exception Spark raises;years * 12andweeks * 7products overflowintindependently of one another.make_intervalproducesCalendarIntervalType, which Comet's columnar layer gained support for in #4898. With that in place the dispatcher can carry its output.The trade-off against #5039: codegen dispatch gives exact Spark semantics for free but runs JVM code per batch, so it is slower than a native kernel. A native implementation is the better end state if it can match Spark on the decimal scaling and both ANSI paths. This PR is the low-risk option and can be superseded.
What changes are included in this PR?
CometMakeIntervalcodegen-dispatch serde indatetime.scala, registered inQueryPlanSerde'stemporalExpressionsmap.make_interval.sqlandmake_interval_ansi.sqlComet SQL file tests.make_intervalrow in the Supported Spark Expressions guide.How are these changes tested?
New Comet SQL file tests run each query through both Spark and Comet, verify the results match, and verify Comet executes the expression through the dispatcher rather than falling back.
make_interval.sql(non-ANSI, sofailOnErroris false) covers:MakeInterval's auxiliary constructors;MakeIntervalisNullIntolerant, soNullPropagationrewrites any call with a literal NULL argument into a null interval literal that never reaches Comet. That literal currently fails natively, tracked separately by Null CalendarInterval literal throws native exception instead of evaluating to null #5058.years * 12exceeds the int range.make_interval_ansi.sqlpinsspark.sql.ansi.enabled=truesofailOnErroris true, and asserts the overflow raises the same error Spark does for both theyears * 12andweeks * 7products. Each error query sits alongside valid-input queries in the same file so the case cannot pass vacuously through a fallback.Verified green on Spark 3.5 and 4.1 locally, along with the pre-existing
calendar_interval.sqlfixture and theexplain comettest inCometExpressionSuite, both of which usemake_interval.