Spell the grouped convex hull as ST_Collect_Agg and align the documented queries

[james-willis]

Q5 computes a convex hull over each customer-month's dropoff locations. This changes how that aggregation is spelled in the base dialect, and fixes two ways the documented queries had drifted from the executable ones.

1. The base dialect spells the hull as a scalar over an array

The base class is documented as the Sedona/Spark SQL dialect, but spelled Q5 as ST_Collect(ARRAY_AGG(...)), while SedonaDBSpatialBenchBenchmark overrode Q5 solely to spell it as the native aggregate ST_Collect_Agg. Sedona Spark ships that aggregate too — registered in st_aggregates.scala as call_udf("ST_Collect_Agg", geometry) — so the base can say it directly and SedonaDB needs no override. ARRAY_AGG also builds a full array of geometries per group before ST_Collect runs, where a native aggregate accumulates incrementally.

DuckDB inherits the base Q5 and has no ST_Collect_Agg, so the ARRAY_AGG spelling moves into a DuckDB override rather than disappearing:

class	Q5 spelling	change
SpatialBenchBenchmark (base)	ST_Collect_Agg	changed
DuckDBSpatialBenchBenchmark	ST_Collect(ARRAY_AGG(…))	override added — same SQL the base had
SedonaDBSpatialBenchBenchmark	inherits base	override removed, class deleted
DatabricksSpatialBenchBenchmark	ST_Union_Agg	unchanged (comment reworded)

SedonaDBSpatialBenchBenchmark is deleted

With the override gone the class had no queries left. Its docstring would have claimed it existed to name the dialect, but dialect() is never called anywhere in the repo — every mention is a definition, a docstring, or an unrelated local variable. The harness takes its engine name from a string literal at run_benchmark.py:393 and its display label from separate maps in summarize_results.py and verify_results.py.

So it is repointed to the base in the two places that referenced it: print_queries.py's CLI map (keeping the "SedonaDB" key, so print_queries.py SedonaDB still works) and run_benchmark.py's dialect map. This widens the diff into benchmark/run_benchmark.py, which this PR does not otherwise touch — worth flagging, though the class is only deletable because the base now carries the aggregate spelling, so it is a direct consequence of the main change rather than an unrelated cleanup.

DuckDB and Databricks keep their classes, because they still override real queries (DuckDB q5 + q12; Databricks q5, q7, q12). Only the empty one goes, so this is not an arbitrary asymmetry.

Verified mechanically rather than assumed: the query dict get_sql_queries("sedonadb") returns is byte-identical before and after — same sha256 across all twelve queries.

dialect() is dead code on the remaining classes too. That predates this PR and is left alone rather than widening scope again; possible follow-up if maintainers want it.

This requires Sedona >= 1.8.1 for the base dialect

Measured, one Spark 3.5.3 session per version:

Sedona Spark	ST_Collect(ARRAY_AGG(…))	ST_Collect_Agg(…)
1.7.2	works	UNRESOLVED_ROUTINE
1.8.0	works	UNRESOLVED_ROUTINE
1.8.1	works	works
1.9.1	works	works

Stating that plainly because it is a real floor. In this repo nothing pins a Sedona version, docs/requirements.txt is an unpinned apache-sedona[db] that resolves to 1.9.1, and there is no Sedona Spark runner in run_benchmark.py — so no engine the harness runs is affected. DuckDB in particular is unaffected, because it now carries its own override.

2. The documented Q5 could not run at all

The Queries page and notebooks/queries.ipynb spelled it ST_Collect(ST_GeomFromWKB(...)) inside an sd.sql(...) call. SedonaDB has no ST_Collect, so the documented query failed at planning time with Invalid function 'st_collect'. The geography suite's Q5 already uses ST_Collect_Agg; this brings the geometry pages in line.

3. The documented Q7 used a different metre constant

It multiplied by 111111 where every executable query divides by 0.000009. Those are not the same number — 1/0.000009 = 111111.111… — so the documented Q7 was off by exactly 1e-6 relative, which is not inside the CI tolerance of rtol=1e-6 but exactly on it: detour_ratio reached 1.000001e-06 and passed only because numpy.isclose adds an atol term.

111111 appeared only in those three doc lines. / 0.000009 is the convention in the executable SQL (print_queries.py:165,328), every other engine (geopandas_queries.py:306, spatial_polars.py:331-333, pycanopy_queries.py:260) and wherobots/benchmark-dashboard; the repo documents it in docs/geography-queries.md:37 and its table row | Unit conversion in SQL | / 0.000009 | none needed |; and it is structural — the radii 0.45 (50 km), 0.045 (5 km) and 0.0045 (500 m) are all exact multiples of 0.000009.

Verification

No engine's executable SQL changes except the base dialect's. Comparing every query before and after with comments and whitespace normalized away:

SedonaDB             NONE          DuckDB Q5 byte-identical to today's:  True
DuckDB               NONE          SedonaDB Q5 unchanged, now from base: True
Databricks           NONE          sedonadb query dict sha256 unchanged: True
SedonaSpark (base)   ['q5']

The SedonaDB and DuckDB rows are taken through the harness's own get_sql_queries() entry point, so they exercise the repointed maps rather than the classes directly.

Every Q5 spelling agrees, by value, on real SF1 data — all bitwise identical to each other and matching benchmark/answers/sf1/q5.csv:

dialect / engine	result
new base ST_Collect_Agg — Sedona Spark 1.9.1	match
old base ARRAY_AGG — Sedona Spark 1.9.1	match
Databricks ST_Union_Agg form — via Sedona Spark, which has that function	match
SedonaDB 0.4.1, its own dialect	match
DuckDB 1.5.5 + spatial, its own dialect	match

All twelve documented queries were executed against SedonaDB 0.4.1 at SF1 over all six tables, through both the documented import sedona.db and the standalone sedonadb package. Before, eleven ran and Q5 was the only failure; now all twelve run, and all twelve match the committed answers under rtol=1e-6 with no atol slack — which Q7 previously did not.

Q7 against answers/sf1/q7.csv:

column	before (* 111111)	after (/ 0.000009)
line_distance_m	1.000000e-06	0.0
detour_ratio	1.000001e-06	0.0

It is now bit-for-bit identical to the committed answer, and the documented Q7 is character-identical to the executable one.

Notebook outputs

Handled per cell. Q5's is left exactly as it was — re-executing the corrected cell reproduces it byte for byte, so the committed output was already the right data attached to code that could not produce it. Q7's is regenerated from a real execution, since the constant moves the values in the sixth significant figure; the new values match answers/sf1/q7.csv exactly, the old ones did not. No other cell touched, and the notebook JSON round-trips with no formatting churn.

benchmark/answers/ is not modified. This supersedes #141, whose content is folded in here.

CI status: the red checks are not from this change

The benchmark workflow shows failures, all of them GeoPandas or Spatial Polars at SF10. They are pre-existing infrastructure timeouts:

• Every failure is a cancelled step, not an assertion or an error — the SF10 runners time out.
• All 20 of the last 20 main runs of benchmark.yml failed, and the failing set drifts between runs (three recent main runs failed 7, 10 and 12 jobs, with membership shifting), which is the signature of flaky infrastructure rather than a deterministic break.
• Neither engine reads the file this PR changes. Only DuckDB and SedonaDB call get_sql_queries() (run_benchmark.py:598,608); GeoPandas, Spatial Polars and PyCanopy each use {f"q{i}": None} and run their own DataFrame implementations.

In fairness this run failed more SF10 jobs than the main run I compared against (17 vs 10), and seven of them were green in that particular baseline — but they are all in the same unstable GeoPandas/Spatial Polars SF10 set, and none of them touch the changed code path.

The checks that actually exercise this change are all green. DuckDB and SedonaDB are the only consumers of print_queries.py, and every one of their jobs passes — including the two that matter most here:

job	SF1	SF10
DuckDB q5	pass	pass
SedonaDB q5	pass	pass

All 24 SedonaDB jobs and all 24 DuckDB jobs pass, at both scale factors.

---

🤖 Generated with Claude Code

[jiayuasu]

Could we refresh the sample output below, and the matching block in queries.zh.md, to go with this conversion? The notebook now shows 11111.126052681648 / 8.99998789734414e-9 for the first row, while both rendered pages still show the old * 111111 values.