feat!: assign extension anchors per plan, not per registry

[nielspardon]

Removes the extension-merging half of #207 as well (the schema re-inference half is
untouched).

Problem

ExtensionRegistry hands out function_anchor / extension_urn_anchor values at
registration time, and builders stamp those registry-global numbers into plans.
Anchors are plan-local in Substrait, so this caused two problems.

Plans were not reproducible. A single-add plan emitted function_anchor: 284
against the default extension set and 4 against a minimal one — the value encoded
how many functions the other default YAMLs defined and the order the
functions*.yaml glob happened to return them (filesystem order, not sorted). A
substrait-extensions bump, or a different machine, shifted every anchor.

Extending a plan built elsewhere silently corrupted it. merge_extension_urns /
merge_extension_declarations dedupe by identity and their docstrings state
"Assumes that there are no collisions", with nothing enforcing it. Given a plan
already using anchor 10/284 for different entities:

extension_urn_anchor: 10  urn: "extension:acme:custom"
extension_urn_anchor: 10  urn: "extension:io.substrait:functions_arithmetic"
function_anchor: 284  name: "acme_thing:i64"   extension_urn_reference: 10
function_anchor: 284  name: "add:i64_i64"      extension_urn_reference: 10

function_reference: 284 is now ambiguous, and no error is raised.

Approach

ExtensionCollector owns those anchors for the duration of one build. Function
references are allocated on first use from 1; URN anchors are derived at emit time,
since nothing outside SimpleExtensionDeclaration refers to one. This follows
substrait-java's io.substrait.extension.ExtensionCollector, including its
first-use numbering and its deferral of URN anchors.

The collector reaches builders through a contextvar, as the builders' other
per-build state already does (_rel_anchor_counter, outer_schemas,
anchor_scope). Three seams carry the change: build_scoped wrapping each
resolver, plus _bind (plans) and resolve_expression (expressions) for adopting
inputs.

An incoming materialized plan has its declarations read back to (urn, name)
identities and its references re-derived rather than trusted, so two
independently numbered inputs cannot disagree about what a reference means. This is
load-bearing rather than defensive: the SQL translator builds a set operation's two
sides as separate plans before merging them, and both number from 1.

Two deliberate points of permissiveness:

• Identities come off the declaration, not a catalog lookup, so a plan naming
  functions absent from the registry still round-trips.
• Declarations too under-specified to re-derive are preserved verbatim. pyarrow's
  serialize_expressions emits a bare extension_function { name: "add" } — no
  anchor, no URN, no extension_urns entry. Anchor 0 is not a reference any
  expression can name under the spec's 1-based numbering, so re-assigning it would
  desynchronize the declaration from expressions that still say 0.

ExtensionRegistry becomes a pure catalog. Its urn→function mapping, signature
matching and extension-relation registration are unchanged.

Result

Anchors are now dense and plan-local:

extension_urn_anchor: 1  urn: "extension:io.substrait:functions_arithmetic"
function_anchor: 1       name: "add:i64_i64"

Byte-identical across a minimal vs. the full default registry, and across repeated
builds of the same frame.

Because the collector accumulates once per build, per-level extension merging is
gone rather than optimized. Measured against a main worktree over an N-verb
project chain:

N	main (merge calls / declarations scanned)	this branch
5	10 / 20	0 / 0
10	20 / 50	0 / 0
20	40 / 110	0 / 0
40	80 / 230	0 / 0

Wall-clock build time is essentially unchanged (59 → 61 ms at N=40): schema
re-inference still dominates, which is #207's other half and out of scope here.

Examples were diffed against main: pyarrow_example and duckdb_example are
byte-identical; builder_example and dataframe_example differ only in anchor
values (89/284/479 → 1–4) with references rewritten consistently.

BREAKING CHANGE: extension anchors are now numbered per plan rather than per
registry, so plans compared byte-for-byte against output from an earlier release
will differ. Anchors are plan-local by spec, so plan semantics are unaffected.
ExtensionRegistry.lookup_urn and FunctionEntry.anchor are deprecated: both now
emit DeprecationWarning and no longer return registry-global anchors. Use
ExtensionRegistry.has_urn() or ExtensionRegistry.urns() to test URN membership,
and identify a function by (entry.urn, str(entry)) instead of by anchor.

Closes #236

🤖 Generated with AI