You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Dataset] TestAceroSubstraitConsumer#testRunQueryLocalFiles fails against Arrow C++ main #1454
Describe the bug, including details regarding any error messages, version, and platform.
The nightly RC workflow builds the JNI libraries from Arrow C++ main. Since Oct 15, every nightly run that reached the arrow-dataset tests has failed the Binaries job on:
TestAceroSubstraitConsumer.testRunQueryLocalFiles:94 expected: <Schema<ID: Int(32, true), NAME: Utf8>> but was: <Schema<ID: Int(32, true) not null, NAME: Utf8>>
The Substrait plan in dataset/src/test/resources/substrait/local_files_users.json declares ID as NULLABILITY_REQUIRED in the read relation's base schema. The Parquet file the test reads has an optional id column (user.avsc declares it as ["int", "null"]), and the test expects a nullable ID.
apache/arrow#51691 (GH-51231), merged into Arrow C++ main on Oct 9, makes Acero's project node keep the input nullability of direct field references. The plan's project relation now returns ID as not null, as the plan declares. 26.0.1 doesn't include that change, so once Arrow C++ 27.0.0 is released, push and pull request runs will fail the same way.
The plan should declare ID as nullable, which matches the file.
Activity