Repository navigation
perf: try bucket sort - #26587
perf: try bucket sort#26587rluvaton wants to merge 2 commits into
Conversation
Adds datafusion.execution.sort_bucket_sort_enabled, sort_bucket_count, sort_bucket_target_bytes, sort_bucket_sample_rows and sort_bucket_mode, used by the bucket sort of ExternalSorter added next.
Instead of sorting every buffered batch and k-way merging the sorted runs, rows are scattered on arrival into key-range buckets chosen from a sample of the sort keys, every bucket is sorted on its own (only its keys: the rows are gathered once in sorted order) and the buckets are emitted in key order. When memory runs out, the rows of every range are appended, unsorted, to that range's spill file, so spilling sorts nothing and reading back needs no merge; a range too large for memory falls back to a nested merge sort. Enabled by datafusion.execution.sort_bucket_sort_enabled. The merge sort still sorts an input too small for the bucket sort, a first sort key of an unsupported type, and an input arriving in reverse sort order once it has to spill: boundaries chosen from its first rows would hold none of the next ones. Tests checking the merge sort's own spill behavior (spill file counts, multi-level merge re-spills) now disable the bucket sort, and a fuzz test measures spilled bytes instead of estimating them from the spill count.
|
run benchmark sort |
|
run benchmarks sort_tpch sort_tpch10 tpch tpch10 tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark tpcds
env:
prefer_hash_join: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark sort_tpch
env:
prefer_hash_join: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark tpch
env:
prefer_hash_join: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark tpch10
env:
prefer_hash_join: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark sort_tpch10
env:
prefer_hash_join: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark sort
env:
BENCH_FILTER: "sort_axis/sort.*.1M.High.cardinality.Unsorted..100cols"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark tpch
env:
prefer_hash_join: "false"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark sort_tpch
env:
prefer_hash_join: "false"CPU Details (lscpu)Details
Resource Usagesort_tpch — base (merge-base)
sort_tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark tpch10
env:
prefer_hash_join: "false"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark tpcds
env:
prefer_hash_join: "false"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark sort_tpch10
env:
prefer_hash_join: "false"CPU Details (lscpu)Details
Resource Usagesort_tpch10 — base (merge-base)
sort_tpch10 — branch
File an issue against this benchmark runner |
|
run benchmark external_aggr |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark external_aggrResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark external_aggrCPU Details (lscpu)Details
Memory Pool PeaksPeak Base:
Pool accounting vs. process RSS Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
Resource Usageexternal_aggr — base (merge-base)
external_aggr — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sort-bucket (6fdf9a9) to e12901d (merge-base) diff Run configurationrun benchmark sort
env:
BENCH_FILTER: "sort_axis/sort.*.1M.High.cardinality.Unsorted..100cols"CPU Details (lscpu)Details
Resource Usagesort — base (merge-base)
sort — branch
File an issue against this benchmark runner |
|
bad perf |
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
No description provided.