Skip to content

feat: [branch-1.0] support explode_outer (#5192) - #5261

Merged
andygrove merged 1 commit into
apache:branch-1.0from
andygrove:backport-5192-branch-1.0
Aug 4, 2026
Merged

feat: [branch-1.0] support explode_outer (#5192)#5261
andygrove merged 1 commit into
apache:branch-1.0from
andygrove:backport-5192-branch-1.0

Conversation

@andygrove

Copy link
Copy Markdown
Member

Backport of #5192 to branch-1.0.

Cherry-picked cleanly from 2af9cec38b1f68550c3e13c1109f3c7502081e84 with no conflicts and no modifications — the diff is identical to the original PR.

Which issue does this PR close?

Closes #2838 and #5224 on branch-1.0.

Rationale for this change

Comet previously routed GenerateExec with outer = true back to Spark (Incompatible) because DataFusion's UnnestExec with preserve_nulls = true emits one null row for a NULL list but drops rows whose list is empty. Spark's explode_outer / posexplode_outer must emit one null row for both cases, so anything containing empty arrays fell back to JVM whole-stage codegen.

What changes are included in this PR?

Native:

  • New ListEmptyToNullExpr (native/core/src/execution/expressions/list_empty_to_null.rs) rewrites a List<T> to mark every empty row as null while preserving the original offsets, values, and column name.
  • planner.rs wraps the array child with ListEmptyToNullExpr when explode.outer is true, before positions are computed and before the projection feeds UnnestExec. ListPositionsExpr inherits the modified null bitmap so pos and value stay aligned for posexplode_outer.

Serde:

  • CometExplodeExec.getSupportLevel no longer returns Incompatible for op.outer. Unsupported cases (maps, non-deterministic generators, multi-input generators, COMET_EXEC_EXPLODE_ENABLED = false) still fall back to Spark whole-stage codegen through the standard Unsupported path.

Tests:

  • Un-ignored explode_outer with empty array, explode_outer with nullable projected column, explode_outer with mixed null, empty, and non-empty arrays in CometGenerateExecSuite.
  • Dropped the WHERE id != 4 workaround and stale allowIncompatible Config: directive in posexplode.sql.
  • Added sql-tests/expressions/array/explode.sql covering explode / explode_outer (plus LATERAL VIEW and LATERAL VIEW OUTER) across every primitive element type, nested array<array<int>>, array<struct>, NULLs, literal arrays, empty tables, and an expect_fallback for map input.

Are these changes tested?

Yes — same tests as the original PR. Verified locally on branch-1.0: cargo build succeeds and all 18 list_empty_to_null / list_positions / related Rust unit tests pass. branch-1.0 pins the same DataFusion version (54.1.0) as main, so no API adaptation was needed.

Are there any user-facing changes?

explode_outer and posexplode_outer now run natively without requiring spark.comet.operator.GenerateExec.allowIncompatible = true. No behavioral change for explode / posexplode.

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove

@andygrove
andygrove merged commit 3a7a2c4 into apache:branch-1.0 Aug 4, 2026
69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants