Repository navigation
Conversation
…umn (apache#6743) * fix: compare every row when arrays_overlap gets a scalar beside a column When one argument was a scalar, SparkArraysOverlap turned it into a one-row list while the column kept its full length, and returned row 0's result as a scalar. A constant array beside an array column therefore answered every row with the first row's result, or panicked with an index out of bounds when the constant was the second argument. Keep the scalar as a one-row list that every output row reads, and return an array result when the other argument is an array. Repeating the scalar to the batch length instead could overflow the list's 32-bit offsets for a long constant. Two scalar arguments still produce a scalar result. Assisted-by: Claude Code (claude-opus-5-5) * test: reuse the arrays_overlap fixture tables for the scalar case Write test_arrays_overlap and test_overlap_struct as one file each, so their rows share a batch. The existing constant-second queries then cover the scalar-beside-column path, and a constant-first query with answers that change by row, plus a struct query in both orders, complete it. This replaces the separate test_overlap_batch table and the flat and struct Rust tests it duplicated. The long-constant and two-scalar Rust tests stay, since a fixture cannot reach them. Assisted-by: Claude Code (claude-opus-5-5) (cherry picked from commit c39eb71)
github-actions
Bot
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Backport of #6743 to
branch-1.2. Closes #6742 on this branch.Rationale for this change
When one argument of
arrays_overlapis a scalar and the other is an array column, the native kernel returns row 0's result for the whole batch, or panics when the constant is the second argument (#6742, a correctness bug).branch-1.2has the same kernel. See #6743 for the full analysis.What changes are included in this PR?
Cherry-pick of c39eb71 (
git cherry-pick -x) with no conflicts and no adaptations. The diff against the source commit is identical apart from line numbers and blob hashes.SparkArraysOverlapkeeps a scalar argument as a one-row list that every output row reads, and returns an array result when the other argument is an array.Other release branches:
branch-1.2is the newest release branch, so there is no newer one to port to. Whetherbranch-1.1needs a backport is not part of this PR.How are these changes tested?
On this branch, with JDK 17: the 30
arrays_overlapRust unit tests indatafusion-comet-spark-exprpass, including the new ones, andCometSqlFileTestSuiteexpressions/array/arrays_overlap.sqlpasses against a native library built from this branch. The same SQL test fails against thebranch-1.2native library without the fix, so the branch has the bug and the test catches it. I ran the default Spark profile only.