Skip to content

Honor the deterministic flag for Rust UDFs #5249

Description

@andygrove

Follow-up from review of #4459.

CometRustUDF.register accepts a deterministic parameter and carries it in the RustUdfCall proto, but the native side ignores it: ImportedCScalarUdf::try_new hardcodes Volatility::Immutable. A nondeterministic UDF planned as immutable may be constant-folded, evaluated once and reused, or eliminated as a common subexpression.

#4459 closes the hole by rejecting deterministic = false at registration, so the flag cannot silently lie. This issue is about actually supporting it.

The obstacle is lifetime mismatch. The Signature is built once per library load and cached process-wide per (library_path), while determinism is declared per CometRustUDF.register call. Two registrations of the same kernel with different determinism would need different volatility from one cached ScalarUDFImpl.

Options worth weighing:

  • Key the cached adapter on (library_path, name, volatility) rather than on the library alone.
  • Build the ScalarUDFImpl per call site in the planner from the cached kernel, taking volatility from RustUdfCall.deterministic, and keep only the loaded library in the cache.
  • Let the kernel declare its own volatility over the ABI and validate the registration against it, which overlaps with the get_property idea in Add a forward-flexible property slot to the Comet UDF C ABI #5254.

Once this lands, drop the guard in CometRustUDF.register, the note on the deterministic field in expr.proto, the "Immutable functions only" limitation in docs/source/user-guide/latest/rust_udfs.md, and the corresponding note on the CometCScalarUdf trait docs.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions