Skip to content

Keep wide-decimal scalar AVG native with a wider running sum #7116

Open

Description

@sunchao

What problem does this solve?

Keep high-precision ungrouped decimal AVG/TRY_AVG and ever-expanding decimal AVG windows native where Spark retains a wide temporary sum until division. PR #5420 repairs empty partials, sticky overflow, and division scaling, but conservatively falls back when the input precision is at least 28. Aggregate fallback also moves sibling MIN/MAX/COUNT/SUM expressions to Spark.

This is the follow-up requested in #5420 (comment). #6041 established a similar design for decimal SUM: its scalar accumulator keeps an i256 running sum and checks precision at state export/evaluation, while the planner declines ungrouped execution without codegen because Spark's UnsafeRow buffer latches overflow earlier.

Proposed work

  • Give scalar AvgDecimalAccumulator a wide running sum, checking overflow at the boundaries required by Spark's AVG partial state and final division.
  • Keep ungrouped maximum-precision AVG in Spark when codegen is disabled; preserve restrictions for grouped object/sort aggregation and unsafe decimal mixed-engine buffers.
  • Revisit the ever-expanding window guard once the scalar accumulator matches Spark for cancellation after an intermediate overflow.
  • Keep the grouped accumulator's sticky overflow behavior, which models Spark's UnsafeRow aggregation buffer.

Validation

Cover empty/all-null input, cancellation such as DECIMAL(38,38) values 0.6, 0.6, -0.4, partial-state export/merge, ANSI/legacy/TRY_AVG, codegen on/off, shuffled and unshuffled global aggregates, and ever-expanding windows. Include sibling aggregate route assertions and a benchmark comparing the current fallback with the native path. Verify the arithmetic bounds of i256 intermediates, including division scaling.

Activity

  1. tlfua commented on Nov 7, 2026

    @tlfua

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:aggregationHash aggregates, aggregate expressionsenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions