Skip to content

#1390 - configurable backend for group by buckets#2473

Merged
norberttech merged 3 commits into
flow-php:1.xfrom
MrHDOLEK:#1390
Jul 12, 2026
Merged

#1390 - configurable backend for group by buckets#2473
norberttech merged 3 commits into
flow-php:1.xfrom
MrHDOLEK:#1390

Conversation

@MrHDOLEK

@MrHDOLEK MrHDOLEK commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Makes GroupBy aggregation memory-efficient and configurable (issue #1390): keep the fast in-memory path by default, or opt into an external, bounded-memory path.

Resolves: #1390
Resolves: #2472 because this pr resolve this problem.

Change Log


Added

  • Configurable, memory-efficient GroupBy aggregation: ConfigBuilder::aggregationAlgorithm() and the AggregationAlgorithms enum select between MEMORY_AGGREGATION (default, in-memory hash map) and EXTERNAL_AGGREGATION (sorts by the group-by columns and folds, spilling to disk for bounded memory), with aggregationFilesystem() for the spill protocol.
  • GroupBy streams its result in batches instead of materializing every group into a single Rows.

Fixed

Changed

Removed

Deprecated

Security

@codecov

codecov Bot commented Jun 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.62656% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.36%. Comparing base (7080d25) to head (fe3abaa).
⚠️ Report is 1 commits behind head on 1.x.

Additional details and impacted files
@@            Coverage Diff             @@
##                1.x    #2473    +/-   ##
==========================================
  Coverage     86.36%   86.36%            
- Complexity    22901    22958    +57     
==========================================
  Files          1740     1748     +8     
  Lines         70386    70535   +149     
==========================================
+ Hits          60789    60918   +129     
- Misses         9597     9617    +20     
Components Coverage Δ
etl 89.83% <89.62%> (-0.07%) ⬇️
cli 89.40% <ø> (ø)
lib-array-dot 81.44% <ø> (ø)
lib-azure-sdk 64.44% <ø> (ø)
lib-doctrine-dbal-bulk 93.61% <ø> (ø)
lib-filesystem 86.04% <ø> (ø)
lib-types 90.18% <ø> (ø)
lib-parquet 70.23% <ø> (ø)
lib-parquet-viewer 82.26% <ø> (ø)
lib-snappy 90.26% <ø> (+1.32%) ⬆️
lib-dremel 0.00% <ø> (ø)
lib-postgresql 88.62% <ø> (ø)
lib-telemetry 86.58% <ø> (ø)
bridge-filesystem-async-aws 92.74% <ø> (ø)
bridge-filesystem-azure 90.45% <ø> (ø)
bridge-monolog-http 96.82% <ø> (ø)
bridge-monolog-telemetry 94.79% <ø> (ø)
bridge-openapi-specification 92.07% <ø> (ø)
symfony-http-foundation 78.57% <ø> (ø)
bridge-psr18-telemetry 100.00% <ø> (ø)
bridge-psr3-telemetry 98.95% <ø> (ø)
bridge-psr7-telemetry 100.00% <ø> (ø)
bridge-telemetry-otlp 90.41% <ø> (ø)
bridge-symfony-http-foundation-telemetry 92.85% <ø> (ø)
bridge-symfony-filesystem-bundle 90.66% <ø> (ø)
bridge-symfony-filesystem-cache 98.18% <ø> (ø)
bridge-symfony-postgresql-bundle 93.39% <ø> (ø)
bridge-symfony-postgresql-cache 94.41% <ø> (ø)
bridge-symfony-postgresql-messenger 98.80% <ø> (ø)
bridge-symfony-postgresql-session 93.65% <ø> (ø)
bridge-symfony-telemetry-bundle 90.10% <ø> (ø)
adapter-chartjs 84.05% <ø> (ø)
adapter-csv 91.66% <ø> (ø)
adapter-doctrine 90.79% <ø> (ø)
adapter-google-sheet 99.18% <ø> (ø)
adapter-http 73.23% <ø> (ø)
adapter-json 88.71% <ø> (ø)
adapter-logger 50.00% <ø> (ø)
adapter-parquet 83.16% <ø> (ø)
adapter-text 74.13% <ø> (ø)
adapter-xml 86.58% <ø> (ø)
adapter-avro 0.00% <ø> (ø)
adapter-excel 94.21% <ø> (ø)
adapter-postgresql 91.06% <ø> (ø)
adapter-seal 93.87% <ø> (ø)
bridge-phpunit-postgresql 75.30% <ø> (ø)
bridge-phpunit-telemetry 87.32% <ø> (ø)
bridge-phpstan-types 0.00% <ø> (ø)
bridge-postgresql-valinor 100.00% <ø> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-official

This comment has been minimized.

Comment thread src/core/etl/src/Flow/ETL/GroupBy/Storage/GroupEntry.php Outdated
Comment thread src/core/etl/src/Flow/ETL/Config/ConfigBuilder.php Outdated
Comment thread src/core/etl/src/Flow/ETL/Config/ConfigBuilder.php Outdated
Comment thread src/core/etl/src/Flow/ETL/GroupBy/Storage/AggregationStorage.php Outdated
Comment thread src/core/etl/src/Flow/ETL/Function/MergeableAggregatingFunction.php Outdated
@github-actions github-actions Bot added size: L and removed size: XL labels Jul 6, 2026
@MrHDOLEK MrHDOLEK requested a review from norberttech July 6, 2026 20:26
@github-actions github-actions Bot added size: XL and removed size: L labels Jul 7, 2026
Comment thread src/core/etl/src/Flow/ETL/GroupBy/HashAggregation.php Outdated
Comment thread src/core/etl/src/Flow/ETL/Config/Aggregation/AggregationConfigBuilder.php Outdated
Comment thread src/core/etl/src/Flow/ETL/Processor/GroupByProcessor.php Outdated
@norberttech norberttech added this to the 0.42.0 milestone Jul 11, 2026
@norberttech norberttech added the hackaton-07-2026 A first Flow Hackaton with Stloyd and MrHDOLEK 🎉 label Jul 11, 2026
@MrHDOLEK MrHDOLEK force-pushed the #1390 branch 4 times, most recently from c81915c to 83bf6e7 Compare July 11, 2026 14:11
Comment thread src/core/etl/src/Flow/ETL/DataFrame/GroupedDataFrame.php Outdated
Comment thread src/core/etl/src/Flow/ETL/Sort/ExternalSortFactory.php Outdated
@MrHDOLEK MrHDOLEK force-pushed the #1390 branch 2 times, most recently from ef4fd15 to 6d769f2 Compare July 12, 2026 08:34
@norberttech norberttech changed the title #1390 - configurable storage for aggregating functions #1390 - configurable backend for group by buckets Jul 12, 2026
@MrHDOLEK MrHDOLEK requested a review from norberttech July 12, 2026 11:27
@norberttech norberttech merged commit 49d116b into flow-php:1.x Jul 12, 2026
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hackaton-07-2026 A first Flow Hackaton with Stloyd and MrHDOLEK 🎉 size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal]: Bounded-memory GroupBy aggregation (streaming merge + streaming result) Storage for Aggregating Functions

2 participants