Skip to content

[Dataset] TestAceroSubstraitConsumer#testRunQueryLocalFiles fails against Arrow C++ main #1454

Closed

Description

@jbonofre

Describe the bug, including details regarding any error messages, version, and platform.

The nightly RC workflow builds the JNI libraries from Arrow C++ main. Since Oct 15, every nightly run that reached the arrow-dataset tests has failed the Binaries job on:

TestAceroSubstraitConsumer.testRunQueryLocalFiles:94 expected: <Schema<ID: Int(32, true), NAME: Utf8>> but was: <Schema<ID: Int(32, true) not null, NAME: Utf8>>

Latest: https://github.com/apache/arrow-java/actions/runs/42539841523/job/128913506709. Push and pull request runs still pass because they build against the latest Arrow C++ release (26.0.1).

The Substrait plan in dataset/src/test/resources/substrait/local_files_users.json declares ID as NULLABILITY_REQUIRED in the read relation's base schema. The Parquet file the test reads has an optional id column (user.avsc declares it as ["int", "null"]), and the test expects a nullable ID.

apache/arrow#51691 (GH-51231), merged into Arrow C++ main on Oct 9, makes Acero's project node keep the input nullability of direct field references. The plan's project relation now returns ID as not null, as the plan declares. 26.0.1 doesn't include that change, so once Arrow C++ 27.0.0 is released, push and pull request runs will fail the same way.

The plan should declare ID as nullable, which matches the file.

Activity

  1. added this to the 20.0.0 milestone on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions