Skip to content

Wide-decimal overflow reports a different value than Spark #5211

Description

@peterxcli

Describe the bug

In ANSI mode, Comet's wide-decimal arithmetic overflow reports a different value parameter from Spark. PR #5169 fixes the exception type and propagates the error class, SQLSTATE, and query context, but Comet still formats the rescaled i256 result while Spark formats its pre-toPrecision Decimal.

This is a follow-up to #5072.

Steps to reproduce

Using Spark 4.1.2, run the query once with Comet disabled and once with Comet enabled:

SET spark.sql.ansi.enabled = true;

CREATE TABLE tbl (_1 DECIMAL(20, 0)) USING PARQUET;
INSERT INTO tbl VALUES (11000000000000000000);

SELECT _1 * _1 FROM tbl;

Spark reports:

[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003

Comet reports:

[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION] 121000000000000000000000000000000000000.000000 cannot be represented as Decimal(38, 6). If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error, and return NULL instead. SQLSTATE: 22003

Expected behavior

Comet should match Spark's value parameter and report 121000000000000000000000000000000000000 without the output-scale suffix.

Additional context

The regression in spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala added by #5169 documents the current difference: Spark formats its pre-toPrecision Decimal, while Comet formats the rescaled i256 value. The test intentionally compares the structured error fields and query context without comparing this differing message parameter.

Metadata

Metadata

Assignees

Labels

good first issueGood for newcomerspriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions