[adriangb]

Which issue does this PR close?

• Closes Allow extension PhysicalExprs to decode via a per-type registry instead of PhysicalExtensionCodec #24626.

Rationale for this change

The user-visible problem

A library that ships its own PhysicalExpr has to write a PhysicalExtensionCodec to serialize it, in a file separate from the expression, and two such libraries on one session have to compose their codecs by hand. PhysicalExtensionExprNode carries an opaque payload and the encoded children but no type discriminator, so the codec is the discriminator: ComposedPhysicalExtensionCodec writes the position of the encoding codec into the payload bytes, the decoding side must register the same codecs in the same order, and a name collision between two independent crates cannot be detected.

A decoder also had no way to reach session state. PhysicalExprDecodeCtx had schema() and decode() but nothing like ExecutionPlanDecodeCtx::task_ctx(), so an extension expression could not resolve a UDF from the function registry or read session configuration while decoding.

Where this sits in the bigger picture

Physical-plan serialization is moving from central downcast_ref chains in datafusion-proto to per-type hooks that live next to each type: try_to_proto (a trait method) plus a try_from_proto constructor. #22418 did this for built-in PhysicalExprs and #23494 for ExecutionPlans. Third-party types could use the encode half but not the decode half. This PR closes the loop for PhysicalExpr; #24631 does the same for ExecutionPlan with the same shape and names (Extension*FromProto pairing the wire name with the constructor, register_*/decode_* facades over one shared registry, attached through SessionConfig::with_extension).

#24631 (merged) added the shared ProtoDecoderRegistry that both kinds register into. #26353 (open) adds SessionConfig::update_extension, so that several libraries can add to the session's registry without replacing each other's decoders. This PR does not depend on it.

How this relates to PhysicalExtensionCodec

The codec is not deprecated and this PR does not touch ComposedPhysicalExtensionCodec. The decode rule is:

• A PhysicalExtensionExprNode with an expr_name the decoding session has registered decodes through the expression's own ExtensionExprFromProto::try_from_proto. A failure there is fatal: falling through to the codec would let it wrongly decode a payload it never wrote.
• Anything else (no name, or a name this session does not know) takes PhysicalExtensionCodec::try_decode_expr exactly as before. If the codec also fails, the error names the missing registration and lists what the session does know.

The codec keeps UDF/UDAF/UDWF payloads (a function is an instance, not a type), anything not yet migrated, and the released FFI_PhysicalExtensionCodec that datafusion-python exposes.

How FFI will work

The registry stores decoders as a private fn pointer; the public surface is ExtensionExprFromProto, extension_expr_node and the facade functions. The storage can later grow to carry an FFI vtable and private data without breaking anything shipped here. PhysicalExprNode crosses an ABI as prost bytes, the pattern datafusion-ffi already uses for Statistics.

What changes are included in this PR?

Built-in expression serialization is untouched: built-ins keep their inherent try_from_proto, dispatched from their own ExprType variant, and no built-in code or caller changes.

• Wire field. PhysicalExtensionExprNode gains optional string expr_name = 3; — additive and proto3-compatible. The codec encode path writes None.
• Session access while decoding. PhysicalExprDecodeCtx::session::<T>() -> Option<&T> returns the session the decode runs under, type-erased because TaskContext lives in a crate that depends on this one. It is backed by a new defaulted PhysicalExprDecode::session() method (returns None); datafusion-proto and ExecutionPlanDecodeCtx supply their TaskContext, so a decoder calls ctx.session::<TaskContext>(). PhysicalExprDecodeCtx is not generic and its constructor is unchanged.
• expr_id stamping. serialize_physical_expr_with_converter now sets the expression's expression_id() on a node returned by try_to_proto when the hook left expr_id empty (a hook that sets its own keeps it). Without this, an identity-bearing extension written with extension_expr_node would lose its identity and defeat DeduplicatingProtoConverter.
• ExtensionExprFromProto (in datafusion_physical_expr::proto). An extension expression implements this one trait, pairing const NAME with fn try_from_proto(node: &PhysicalExprNode, ctx: &PhysicalExprDecodeCtx<'_>). Its try_to_proto is one call — extension_expr_node::<Self>(ctx, payload, self.children()) — which builds the node and stamps Self::NAME, so the encoded name and the registry key cannot drift apart. Built-ins do not implement it, so register_physical_expr::<Column>(..) does not compile (a compile_fail doctest checks this). extension_expr_node is a free function because PhysicalExprEncodeCtx lives in datafusion-physical-expr-common, below the crate that can name the trait.
• Registration goes into the ProtoDecoderRegistry from feat: allow extension ExecutionPlans to decode via a per-type registry instead of PhysicalExtensionCodec #24631: call register_physical_expr::<T>(&mut registry) for every extension expression the session must decode, and attach it with SessionConfig::with_extension. Registering the same type twice is a no-op; a different type under a taken name, or an empty name, is an error. Names are scoped per kind, so a plan and an expression may share a name in one registry. This crate adds register_physical_expr, physical_expr_names, and decode_physical_expr (#[doc(hidden)], internal to datafusion-proto; it reads the name off the node so a caller cannot pair a node with a name it does not carry).
• Re-exports. datafusion::physical_plan::proto re-exports ExtensionExprFromProto, extension_expr_node, register_physical_expr, physical_expr_names, decode_physical_expr, PhysicalExprDecodeCtx and PhysicalExprEncodeCtx, alongside the ExecutionPlan facade and ProtoDecoderRegistry.
• Decode dispatch in datafusion-proto per the rule above (when the codec fallback also fails, the missing registration is added as context on the codec's own error so its kind still reaches callers), a doc pointer on PhysicalExtensionCodec::try_decode_expr, and an upgrade-guide entry in 56.0.0.

Why a separate trait rather than a method on PhysicalExpr

1. An associated constant makes a trait not dyn-compatible (E0038), which breaks every Arc<dyn PhysicalExpr>, so the wire name cannot live on PhysicalExpr.
2. Cargo unifies features across the graph, so a required method on a proto-gated trait is E0046 in a downstream crate that never enabled the feature.
3. With a default, a forgotten decoder becomes a decode-time NotImplemented instead of a compile error.

Decoding is a constructor with no self and can never be dispatched through &dyn PhysicalExpr, so the encode/decode asymmetry is structural.

Session scoping goes through SessionConfig extensions rather than a new field on TaskContext/SessionState, which would force datafusion-physical-expr-common/proto on for everyone.

What is the testing strategy for this PR?

datafusion/proto/tests/cases/plans/expr_registry.rs, 10 tests. A RefusingCodec errors on every method, so any codec involvement is fatal; a codec that can decode the same payload makes the fallback paths distinguishable from mere failure.

• Registry decode end to end with no codec on either side, including reading the session from the decoder.
• Fallback: unnamed or unregistered names reach the codec; a codec failure keeps its own error and names the missing registration.
• Registration rejects collisions and empty names; same type twice is a no-op.
• Nested extension expressions both decode through the registry.
• A failing registry decoder does not fall through to a codec that could decode the node.
• An extension expression resolves a scalar UDF and an aggregate UDF from the reading session via ctx.session::<TaskContext>().
• A whole plan roundtrip through physical_plan_to_bytes_with_extension_codec / physical_plan_from_bytes_with_extension_codec.
• An identity-bearing extension referenced twice keeps its expression_id() under DeduplicatingProtoConverter: shared state is preserved and children decode correctly.
• A registered extension plan and expression with the same wire name roundtrip together in one plan.

Are there any user-facing changes?

Additive. Nobody is affected unless they opt in; there are no breaking API changes. Documented in the 56.0.0 upgrade guide.

Two things worth a reviewer's attention:

• The wire field is additive for binary protobuf but not for the optional JSON serde: generated JSON deserializers reject unknown fields, so a node carrying exprName needs a reader built after this change.
• The codec fallback is only equivalent for nodes written the old way. Once an expression's try_to_proto writes its own named node, a reader that fails to register the type hands that payload to try_decode_expr, and a codec that still recognizes the expression may decode the new payload as the old message. The upgrade guide says to register migrated expressions on every reader before writers start using them.

🤖 Generated with Claude Code

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-physical-expr v55.1.0 (current)
       Built [  26.608s] (current)
     Parsing datafusion-physical-expr v55.1.0 (current)
      Parsed [   0.052s] (current)
    Building datafusion-physical-expr v55.1.0 (baseline)
       Built [  25.593s] (baseline)
     Parsing datafusion-physical-expr v55.1.0 (baseline)
      Parsed [   0.051s] (baseline)
    Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.421s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  54.727s] datafusion-physical-expr
    Building datafusion-physical-expr-common v55.1.0 (current)
       Built [  20.847s] (current)
     Parsing datafusion-physical-expr-common v55.1.0 (current)
      Parsed [   0.020s] (current)
    Building datafusion-physical-expr-common v55.1.0 (baseline)
       Built [  20.955s] (baseline)
     Parsing datafusion-physical-expr-common v55.1.0 (baseline)
      Parsed [   0.020s] (baseline)
    Checking datafusion-physical-expr-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.253s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  42.771s] datafusion-physical-expr-common
    Building datafusion-physical-plan v55.1.0 (current)
       Built [  33.115s] (current)
     Parsing datafusion-physical-plan v55.1.0 (current)
      Parsed [   0.210s] (current)
    Building datafusion-physical-plan v55.1.0 (baseline)
       Built [  32.607s] (baseline)
     Parsing datafusion-physical-plan v55.1.0 (baseline)
      Parsed [   0.207s] (baseline)
    Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.864s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  68.236s] datafusion-physical-plan
    Building datafusion-proto v55.1.0 (current)
       Built [  48.980s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.012s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  50.054s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.146s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [ 100.301s] datafusion-proto
    Building datafusion-proto-models v55.1.0 (current)
       Built [  21.928s] (current)
     Parsing datafusion-proto-models v55.1.0 (current)
      Parsed [   0.146s] (current)
    Building datafusion-proto-models v55.1.0 (baseline)
       Built [  21.779s] (baseline)
     Parsing datafusion-proto-models v55.1.0 (baseline)
      Parsed [   0.148s] (baseline)
    Checking datafusion-proto-models v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   2.419s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field PhysicalExtensionExprNode.expr_name in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2012
  field PhysicalExtensionExprNode.expr_name in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2012

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  47.548s] datafusion-proto-models

[codecov-commenter]

Codecov Report

❌ Patch coverage is 76.19048% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.92%. Comparing base (7e102d7) to head (5dc260a).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	13 Missing ⚠️
...tafusion/physical-expr-common/src/physical_expr.rs	33.33%	3 Missing and 1 partial ⚠️
datafusion/physical-expr/src/proto.rs	91.66%	1 Missing and 2 partials ⚠️
datafusion/physical-plan/src/proto/mod.rs	0.00%	3 Missing ⚠️
datafusion/proto/src/physical_plan/from_proto.rs	97.05%	1 Missing ⚠️
datafusion/proto/src/physical_plan/to_proto.rs	92.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24628      +/-   ##
==========================================
- Coverage   82.92%   82.92%   -0.01%     
==========================================
  Files        1155     1156       +1     
  Lines      458204   458302      +98     
  Branches   458204   458302      +98     
==========================================
+ Hits       379965   380034      +69     
- Misses      55205    55229      +24     
- Partials    23034    23039       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[adriangb]

I put this in draft. Lets sequence behind #24631 to avoid duplication. I will keep it updated though.

[adriangb]

@gabotechs this is ready for review if you are interested

[kosiew]

@adriangb,

Thanks for working on this. The registry-based decoding approach looks good overall, and I appreciate the backward compatibility with the existing codec path.

I found one issue with preserving expression identity when migrating to the new helper. I've also left two minor suggestions for test coverage and documentation.

[kosiew]

The helper leaves expr_id as None, and the serializer returns hook-generated nodes without adding the expression's ID. This means an extension with expression_id() == Some(42) loses its identity when switching to registry encoding, preventing the deduplicating decoder from restoring shared state. Could we stamp the ID at the common serialization boundary when the hook hasn't supplied one, and add a regression test with two references to an identity-bearing extension that verifies shared state and correctly decoded children?

[adriangb]

Good catch. Fixed in 5dc260a. The serialized now sets expression_id() on any node from a hook that leaves expr_id empty. I added a test for deduping by id.

[kosiew]

Could we add a follow-up roundtrip test containing both a registered extension plan and expression with the same wire name? The existing registry tests cover name isolation, but this would also exercise both registration paths together in a real plan.

[adriangb]

also added in 5dc260a

[kosiew]

This comment refers to PhysicalExprRegistry, but the implementation uses ProtoDecoderRegistry. Could we update the name here and regenerate the corresponding comment in prost.rs?

[kosiew]

@adriangb,

Thanks for addressing the earlier review comments. I've rechecked the follow-up changes, and the expression ID preservation, shared-state deduplication, same-name plan/expression roundtrip, and registry documentation corrections all look good. I don't see any remaining correctness blockers.

One non-blocking suggestion: the PR description still references the old generic PhysicalExprDecodeCtx<'a, S>, ExprDecodeSession, a third constructor argument, and a required PhysicalExprFromProto import for built-ins. Could you update the API examples, migration guidance, re-export list, and test count to match the current implementation?

The prerequisite on #24631 and the major-release compatibility considerations still apply. Otherwise, this looks good to me.

[gabotechs]

Didn't give an in-depth review (trusting @kosiew here), but overall looks good and no surprises 👍 thanks @adriangb!

[adriangb]

PR description updated, thanks for pointing it out @kosiew!