Follow-up from review of #4459 (thread).
ImportedCScalarUdf::return_type already calls the kernel through the C ABI, decodes an FFI_ArrowSchema into a full Field, and then throws everything but the DataType away. The planner separately hardcodes the output field as nullable:
let return_field = Arc::new(Field::new(&call.name, kernel_return_type, true));
So a kernel that reports a non-nullable output, or attaches field metadata, has both discarded.
Implementing ScalarUDFImpl::return_field instead would carry the kernel's own Field through to the plan, and would let the planner stop fabricating one.
Worth checking before doing it:
- Spark UDF results are nullable in Spark's own schema, so promising DataFusion a non-nullable field needs to not trip a schema comparison at the Comet boundary or in the shuffle writer.
- The declared-vs-actual return type check in
planner.rs deliberately erases nested nullability, because Spark carries containsNull in the type while the delivered array normalizes children to nullable. Whatever this does with top-level nullability should be consistent with that.
- A test for a kernel returning a non-nullable field, since nothing exercises that today.
Follow-up from review of #4459 (thread).
ImportedCScalarUdf::return_typealready calls the kernel through the C ABI, decodes anFFI_ArrowSchemainto a fullField, and then throws everything but theDataTypeaway. The planner separately hardcodes the output field as nullable:So a kernel that reports a non-nullable output, or attaches field metadata, has both discarded.
Implementing
ScalarUDFImpl::return_fieldinstead would carry the kernel's ownFieldthrough to the plan, and would let the planner stop fabricating one.Worth checking before doing it:
planner.rsdeliberately erases nested nullability, because Spark carriescontainsNullin the type while the delivered array normalizes children to nullable. Whatever this does with top-level nullability should be consistent with that.