perf: add a GroupsAccumulator for array_agg(DISTINCT) by zhuqi-lucas · Pull Request #26357 · apache/datafusion

zhuqi-lucas · 2026-10-15T06:03:52Z

Which issue does this PR close?

Closes Add a GroupsAccumulator for array_agg(DISTINCT) #26347.

Rationale for this change

array_agg(DISTINCT x) had no GroupsAccumulator. Grouped queries fell back to GroupsAccumulatorAdapter, which keeps one DistinctArrayAggAccumulator per group, each with its own hash table and encoded rows. With many groups this dominates memory and CPU.

What changes are included in this PR?

New DistinctArrayAggGroupsAccumulator:

Each distinct value is encoded once with a RowConverter into a Rows buffer shared by all groups, so a value that appears in many groups is stored once.
Groups record (group_index, value_index) pairs, deduplicated in one hash set.
evaluate builds the ListArray with a counting sort by group.
It also covers array_agg(DISTINCT x ORDER BY x), sorting each group by the encoded rows.
EmitTo::First drops values only referenced by emitted groups.

Small refactors: the DISTINCT / ORDER BY check is shared by the row and groups paths (distinct_sort_options), and the one-row-per-list convert_to_state is shared with ArrayAggGroupsAccumulator.

state_compat now compares list results as multisets for DISTINCT without ORDER BY, since the element order is not defined there. The row accumulator returns hash table order.

Are these changes tested?

Unit tests cover dedup across batches, nulls with and without IGNORE NULLS, FILTER, ORDER BY DESC, EmitTo::First, state/merge, convert_to_state, dictionary types, and a differential check against DistinctArrayAggAccumulator.
state_compat passes with the new accumulator included.
array_agg.slt gains grouped cases mixing DISTINCT with count(*), FILTER, ORDER BY and IGNORE NULLS. The existing aggregate slt files pass.

Benchmarks

Measured locally with a criterion benchmark (not included here, happy to add it): one 8192-row Utf8 batch, update_batch plus evaluate, the new accumulator against GroupsAccumulatorAdapter over DistinctArrayAggAccumulator. Apple M-series:

groups	input	groups accumulator	adapter (before)	speedup
100	~25 distinct	154 µs	251 µs	1.6x
100	~7800 distinct	384 µs	969 µs	2.5x
1000	~25 distinct	173 µs	1.42 ms	8.2x
1000	~7800 distinct	382 µs	2.26 ms	5.9x
8192	~25 distinct	279 µs	10.2 ms	36x
8192	~7800 distinct	414 µs	10.4 ms	25x

Live heap after update_batch, measured with a counting global allocator: 65 536 Utf8 rows in 8 batches. size() tracks these within ~10% for both.

groups	input	groups accumulator	adapter (before)
100	~25 distinct	11 KB	720 KB
1000	~7800 distinct	2.2 MB	9.9 MB
8192	~25 distinct	1.7 MB	18.3 MB
65536	~7800 distinct	2.2 MB	100.7 MB

Memory now grows with the number of distinct values instead of the number of groups.

Are there any user-facing changes?

No API changes. array_agg(DISTINCT x) without ORDER BY may return elements in a different (still unspecified) order.

array_agg(DISTINCT x) had no GroupsAccumulator, so grouped queries went through GroupsAccumulatorAdapter with one DistinctArrayAggAccumulator (and its own hash table and encoded rows) per group. DistinctArrayAggGroupsAccumulator encodes each distinct value once with a RowConverter into a Rows buffer shared by all groups, and deduplicates (group_index, value_index) pairs in a single hash set. It also covers array_agg(DISTINCT x ORDER BY x), sorting each group by the encoded rows. state_compat now compares list results as multisets for DISTINCT without ORDER BY, where the element order is not defined. Closes apache#26347.

Copilot

🟡 Changes recommended

Missing DISTINCT instrumentation breaks existing metric expectations, and unnecessary encoding introduces CPU and memory regressions.

3 open findings

What changed in this PR

Adds a native grouped accumulator for array_agg(DISTINCT) to reduce per-group memory and CPU overhead.

Changes:

Shares distinct-value storage across groups and supports ordering and partial emission.
Reuses state-conversion helpers and adjusts unordered DISTINCT result comparisons.
Adds unit and SQL coverage for grouped DISTINCT aggregation.

File	Description
datafusion/sqllogictest/test_files/array_agg.slt	Adds grouped DISTINCT query coverage.
datafusion/functions-aggregate/tests/state_compat.rs	Compares unordered DISTINCT lists as multisets.
datafusion/functions-aggregate/src/array_agg.rs	Implements the native accumulator, shared helpers, and unit tests.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+impl GroupsAccumulator for DistinctArrayAggGroupsAccumulator {
+    fn update_batch(


+        if rows.is_empty() {
+            return Ok(());
+        }
+        self.insert(input, rows)


+        if rows.is_empty() {
+            return Ok(());
+        }
+        self.insert(lists.values(), rows)


…DISTINCT) - Report the DISTINCT submetric from the native groups accumulator, and keep it across a full emit. - Gather filtered or null-skipped rows before encoding them, and encode only the offset span of a sliced state list when merging. - Build the adapter explicitly in the tests that cover its metrics, and give the spilling test enough groups to spill with the native accumulator.

codecov-commenter · 2026-10-15T08:42:05Z

Codecov Report

❌ Patch coverage is 88.88889% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.93%. Comparing base (7f493a5) to head (13b96c3).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/functions-aggregate/src/array_agg.rs	88.61%	8 Missing and 52 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #26357    +/-   ##
========================================
  Coverage   82.93%   82.93%            
========================================
  Files        1154     1154            
  Lines      457851   458368   +517     
  Branches   457851   458368   +517     
========================================
+ Hits       379701   380150   +449     
- Misses      55131    55144    +13     
- Partials    23019    23074    +55

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jayzhan211

Thanks @zhuqi-lucas , overall LGTM

jayzhan211 · 2026-10-16T12:34:46Z

+            + self.values.as_ref().map_or(0, Rows::size)
+            + self.value_map_size
+            + self.pairs.capacity() * size_of::<(u32, u32)>()
+            + self.pair_set.capacity() * (size_of::<(u32, u32)>() + 1)


Suggested change

+ self.pair_set.capacity() * (size_of::<(u32, u32)>() + 1)

+ estimate_memory_size::<(u32, u32)>(self.pair_set.capacity(), 0).unwrap_or_default()

Should we use this one?

Thanks @jayzhan211, yes, that's better. Applied in 5e25666.

kosiew · 2026-10-19T09:36:47Z

run benchmark array_agg_distinct

adriangbot · 2026-10-19T09:40:07Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6264197522-3589-cktpm 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing array-agg-distinct-groups-accumulator (5e25666) to 5009f42 (merge-base) diff

Run configuration

run benchmark array_agg_distinct

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-19T10:01:17Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing array-agg-distinct-groups-accumulator (5e25666) to 5009f42 (merge-base) diff

Run configuration

run benchmark array_agg_distinct

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                     HEAD                                   array-agg-distinct-groups-accumulator
-----                     ----                                   -------------------------------------
array_agg_distinct/Q01    1.01     27.0±0.52ms        ? ?/sec    1.00     26.7±0.40ms        ? ?/sec

Resource Usage

array_agg_distinct — base (merge-base)

Metric	Value
Wall time	335.2s
Peak memory	617.7 MiB
Avg memory	19.4 MiB
CPU user	78.4s
CPU sys	2.6s
Peak spill	0 B

array_agg_distinct — branch

Metric	Value
Wall time	555.3s
Peak memory	605.2 MiB
Avg memory	12.8 MiB
CPU user	72.4s
CPU sys	2.4s
Peak spill	0 B

File an issue against this benchmark runner

kosiew

@zhuqi-lucas,

Thanks for working on this. The native groups accumulator addresses the per-group memory and CPU overhead by sharing encoded values and deduplicating group/value pairs. The implementation also covers ordering, partial emission, state merging, and null handling, with good regression coverage.

I have two non-blocking suggestions around benchmark coverage and documentation. Neither needs to hold up this PR.

LGTM!

…c comment

…oups-accumulator # Conflicts: # datafusion/functions-aggregate/tests/state_compat.rs

…versed (apache#26440) ## Which issue does this PR close? - Closes apache#26439. ## Rationale for this change `array_agg(DISTINCT x ORDER BY x)` returns its values in the wrong order when an earlier aggregate in the same query orders by `x` in the other direction: ```sql SELECT array_agg(x ORDER BY x DESC), array_agg(DISTINCT x ORDER BY x) FROM (VALUES (2), (NULL), (1), (2)) t(x); -- before: [NULL, 2, 2, 1] [NULL, 2, 1] -- after: [NULL, 2, 2, 1] [1, 2, NULL] ``` `string_agg(DISTINCT x, ',' ORDER BY x)` has the same problem, since it uses the same accumulator. When the aggregates need opposite orderings, the planner swaps one of them for its reverse expression, which flips the `ORDER BY` and sets `is_reversed`. The ordered (non-`DISTINCT`) `array_agg` accumulator undoes that at the end, but the `DISTINCT` accumulator sorts its values itself using the flipped sort options and never looks at `is_reversed`. ## What changes are included in this PR? In `ArrayAgg::accumulator`, the `DISTINCT` branch flips the sort options back when `is_reversed` is set, so `DistinctArrayAggAccumulator` sorts in the order the user asked for. ## What is the testing strategy for this PR? New sqllogictest cases: - `array_agg.slt`: the query above (including a NULL, so NULL placement is checked too) and a `GROUP BY` variant. - `aggregate.slt`: `string_agg(DISTINCT ... ORDER BY ...)` next to a `string_agg(... ORDER BY ... DESC)`. All three fail on `main` and pass with this change. The rest of the sqllogictest suite and the `datafusion-functions-aggregate` unit tests pass. ## Are there any user-facing changes? Queries like the one above now return the documented order. No API changes. ## Other notes apache#26357 adds a `GroupsAccumulator` for `array_agg(DISTINCT)`. If it sorts with the expression's sort options, the same `is_reversed` handling is probably needed there too. Co-authored-by: JanakiRamChittineni <341781521+raams95@users.noreply.github.com> Co-authored-by: Qi Zhu <zhuqilucas@gmail.com>

…oups-accumulator Resolve conflict in array_agg.rs: apply the is_reversed handling from apache#26440 inside the shared distinct_sort_options helper so both the DISTINCT Accumulator and the new DISTINCT GroupsAccumulator honor the requested order. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI balanced review requested due to automatic review settings October 15, 2026 06:03

github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Oct 15, 2026

zhuqi-lucas marked this pull request as draft October 15, 2026 06:04

Copilot AI reviewed Oct 15, 2026

View reviewed changes

Copilot started reviewing on behalf of zhuqi-lucas October 15, 2026 07:27

View session

github-actions Bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Oct 15, 2026

zhuqi-lucas marked this pull request as ready for review October 16, 2026 06:04

Merge branch 'main' into array-agg-distinct-groups-accumulator

d29b125

zhuqi-lucas requested review from alamb and kosiew October 16, 2026 07:15

zhuqi-lucas mentioned this pull request Oct 16, 2026

Add a GroupsAccumulator for array_agg with ORDER BY #26409

Open

zhuqi-lucas requested a review from jayzhan211 October 16, 2026 07:18

jayzhan211 approved these changes Oct 16, 2026

View reviewed changes

raams95 mentioned this pull request Oct 17, 2026

fix: respect ORDER BY in array_agg(DISTINCT) when the aggregate is reversed #26440

Merged

Estimate pair_set memory with estimate_memory_size

5e25666

kosiew approved these changes Oct 19, 2026

View reviewed changes

Comment thread datafusion/functions-aggregate/src/array_agg.rs

Comment thread datafusion/functions-aggregate/src/array_agg.rs Outdated

Add a groups benchmark for array_agg(DISTINCT) and fix a misplaced do…

8758c09

…c comment

zhuqi-lucas mentioned this pull request Oct 20, 2026

Amortize state compaction in DistinctArrayAggGroupsAccumulator on EmitTo::First #26512

Open

zhuqi-lucas added 2 commits October 20, 2026 10:46

Merge branch 'main' into array-agg-distinct-groups-accumulator

13b96c3

Merge remote-tracking branch 'apache/main' into array-agg-distinct-gr…

0577519

…oups-accumulator # Conflicts: # datafusion/functions-aggregate/tests/state_compat.rs

alamb enabled auto-merge October 22, 2026 14:54

alamb added this pull request to the merge queue Oct 22, 2026

Merged via the queue into apache:main with commit 290e263 Oct 22, 2026
44 of 45 checks passed

github-actions Bot added the v56.0.0 label Oct 22, 2026

Navigation Menu

Repository navigation

perf: add a GroupsAccumulator for array_agg(DISTINCT) - #26357

perf: add a GroupsAccumulator for array_agg(DISTINCT)#26357
alamb merged 8 commits into
apache:mainfrom
zhuqi-lucas:array-agg-distinct-groups-accumulator

zhuqi-lucas commented Oct 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

codecov-commenter commented Oct 15, 2026 •

edited

Loading

Uh oh!

jayzhan211 left a comment

Uh oh!

jayzhan211 Oct 16, 2026

Uh oh!

jayzhan211 Oct 16, 2026

Uh oh!

zhuqi-lucas Oct 18, 2026

Uh oh!

kosiew commented Oct 19, 2026

Uh oh!

adriangbot commented Oct 19, 2026

Uh oh!

adriangbot commented Oct 19, 2026

Uh oh!

kosiew left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

		impl GroupsAccumulator for DistinctArrayAggGroupsAccumulator {
		fn update_batch(

	+ self.pair_set.capacity() * (size_of::<(u32, u32)>() + 1)
	+ estimate_memory_size::<(u32, u32)>(self.pair_set.capacity(), 0).unwrap_or_default()

Conversation

zhuqi-lucas commented Oct 15, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Benchmarks

Are there any user-facing changes?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

codecov-commenter commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

jayzhan211 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

zhuqi-lucas Oct 18, 2026

Choose a reason for hiding this comment

Uh oh!

kosiew commented Oct 19, 2026

Uh oh!

adriangbot commented Oct 19, 2026

Uh oh!

adriangbot commented Oct 19, 2026

Uh oh!

kosiew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov-commenter commented Oct 15, 2026 •

edited

Loading