Skip to content

[GH-3369] Preserve declared coordinate layouts in WKB output - #3381

Draft
jiayuasu wants to merge 31 commits into
apache:masterfrom
jiayuasu:fix/wkb-output-dimensions
Draft

jiayuasu wants to merge 31 commits into
apache:masterfrom
jiayuasu:fix/wkb-output-dimensions

Conversation

@jiayuasu

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Yes.

Is this PR related to a ticket?

Follow-up to #3369 and #3374. Depends on the shared IO2 layout API and writer changes. The branch includes #3378 as a prerequisite; review only this output step.

Draft pending prerequisite reviews and publication of IO2.

The IO2 work is reviewed in JTS #11 (shared layouts/reader) and JTS #12 (writer). Both belong to the same pending release.

What changes were proposed in this PR?

ST_AsBinary, ST_AsEWKB, and ST_AsHEXEWKB now preserve declared Z/M layouts for empty geometries and all-NaN ordinates. They use the isolated WKB writer's opt-in preservation mode instead of choosing dimensions from the first coordinate. A finite Z later in a line is no longer dropped, and mixed GeometryCollection members keep their individual layouts.

Ordinary XY output, byte order, and SRID behavior remain intact. The public explicit-dimension writer helper keeps its stock JTS return type and behavior. GeoParquet output and explicit 2D adapter writers are unchanged. Collections with no members lack sequence metadata and are written as XY.

How was this patch tested?

Five of six new common tests fail against the prior automatic output behavior. The updated code passes, and two SQL tests cover output after a shuffle/cache and nulls.

Validated the combined Sedona stack at 76f59c947971e06657d01229a4f940863d3e5253 against the local, unpublished IO2 candidate:

  • Full common suite: 1,396 passed.
  • Spark 3.5/Scala 2.12 and Spark 4.1/Scala 2.13: 59 selected Java tests and 265 Scala tests passed per profile, including SQL functions, UDT, collection and reader suites. The separate MySQL/Docker constructor suite was not run.
  • Stock PySpark 3.5.0 and 4.1.1: 96 native/fallback constructor cases, four strict dimension probes, and 144 additional WKB output checks passed.
  • Spark 4.1 output checks also passed with two separate executor JVMs and its bundled JTS 1.20 jar unchanged. Fresh shaded jars contain exact candidate class bytes and none of the removed Sedona helpers.

The IO2 candidate passes 80 focused JTS core tests and 27 isolated-artifact tests against stock JTS 1.20. These are local results; the dependency remains unpublished.

Did this PR include necessary documentation updates?

Updated all three output function pages, including M preservation and the zero-member collection limit.

Signed-off-by: Jia Yu <jiayu@apache.org>
Signed-off-by: Jia Yu <jiayu@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant