[EPIC] Variant on DataFusion #26517

Is your feature request related to a problem or challenge?

OpenSearch is adding a Variant field type to its Parquet-backed indices. The query engine is DataFusion, the writer and Variant kernels are arrow-rs, and the SQL functions are datafusion-contrib/datafusion-variant. Files mix shredded and unshredded layouts within one table, and the same table is read by Spark.

This epic lists the core hooks that work still needs, following the split in #16116: generic extension points in core, Variant-specific implementations in the crate. It sits under #16116 and #21308. Items already tracked elsewhere are linked rather than restated; the new issues are the ones marked new.

Spelling the type in SQL (section 6) already has upstream issues filed; they are linked here so the epic covers the whole core surface Variant needs.

Describe the solution you'd like

1. Read only the leaves a UDF needs

variant_get(v, '$.x') reads and decodes the whole Variant group because the Parquet planner only trusts get_field. Two PRs address this: #25013 (two hooks on ScalarUDFImpl) and draft #26282 (NestedInputAccess capability, also used by get_field, with row group and page pruning on the declared leaves). Our requirement for whichever shape lands: a function must be able to require more than one leaf for one argument (the typed leaf and the value fallback leaf), and pruning must keep any container that holds a fallback row. #26282 does both.

• Review feat: expose UDF field access and input requirements for Parquet pruning #25013 / feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282 against shredded and unshredded test data and help converge on one shape
• Row group and page pruning on a declared leaf, with the fallback guard: Support row group pruning for struct field predicates #20871. feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282 includes it; if the work is split, this is the successor to feat: expose UDF field access and input requirements for Parquet pruning #25013
• datafusion-variant: implement the capability for variant_get once it lands

2. Keep the extension tag through planning

A Variant is a struct whose field metadata carries ARROW:extension:name = arrow.parquet.variant. Two core paths drop it, after which variant_get sees a plain struct and rejects the column.

• new Parquet schema inference clears all field metadata before Schema::try_merge when skip_metadata = true (the default), so any ListingTable over more than one file loses the tag (datasource-parquet/src/file_format.rs, clear_metadata). Reproduced on 54. Proposed fix: keep the ARROW:extension:* keys recursively and drop the rest; the extension keys are the type's identity, not writer annotation.
• CASE / COALESCE build the output Field from the DataType alone: CASE expressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860 (reported on geoarrow.wkb, same mechanism)

3. Tables that mix shredded and unshredded files

When one file stores v as Struct<metadata, value> and another as Struct<metadata, value, typed_value: Struct<x: Int64>>, the merged table schema has typed_value, and the default adapter null-fills it for the unshredded file (physical-expr-adapter/src/schema_rewriter.rs). For an ordinary struct that is correct. For Variant the shredding spec makes value = object, typed_value = null an invalid row, so a compliant reader returns NULL for every field of every row of that file, silently. Reproduced: a two-file table returned 0 of 1024 unshredded rows. Files shredded on different keys fail differently, at open, with target field is non-nullable but missing from source.

The constraint we would like core to hold: a file is read in its own layout and reconstructed, never null-filled into a layout it does not have. Two ways to satisfy it: per-file required_fields answers (#26282 already lets the answer differ per file; to be confirmed whether that alone avoids the null-fill), or a hook for a registered extension type to supply its own per-file adaptation. The Variant adapter itself would live in the crate. Named earlier on #16116 (2025-07-15) and #22908 (2026-09-08).

• new Per-file adaptation for extension types, so an unshredded file is never null-filled into the shredded layout: per-file required_fields answers (feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282) or an extension-type adapter hook, with the Variant adapter in the crate

4. Statistics the optimizer can use

ColumnStatistics is flat, so a nested leaf (shredded or not) and any UDF output have no planner statistics. Two designs are open for the UDF half and one has to be picked; #26282 adds a container-level output_statistics for pruning, which is a third data point.

• Nested and extension-type leaves in Statistics: Nested types and extension types are not supported in Statistics / ColumnStatistics #20707
• UDF output statistics: ScalarUDFImpl::output_statistics (ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307) or ExpressionAnalyzer (Add ExpressionAnalyzer for pluggable expression-level statistics estimation #21122). We implement whichever is chosen; the crate half (typed-leaf min/max, fallback-leaf null count) is the same code either way.

5. Variant value semantics

GROUP BY, DISTINCT, joins and ORDER BY on a Variant column are undefined today. Spark rejects all of them at analysis time. DuckDB 1.5 allows them, but =, GROUP BY, DISTINCT and ORDER BY each apply a different rule (equal numbers of different width compare equal but group apart; ORDER BY is by inner type tag, so 7.0 sorts after 42).

• new, after design discussion A canonical-bytes normalisation kernel in arrow-rs so equal values have equal bytes, and a variant_normalize function in the crate on top of it, so hashing operators work without each inventing a rule. For ORDER BY our lean is to refuse at plan time as Spark does. Issue to follow once the SQL semantics are written up.

6. Spelling VARIANT in SQL

The generic machinery is merged: ExtensionTypeRegistry and DFExtensionType (#20312, DataFusion 54), TypePlanner (#20676, 54), and metadata-preserving CAST (#23169, 55.1.0 via backport #24875). What is missing is the Variant entries. Without them a Variant prints as raw struct bytes in datafusion-cli, and CREATE TABLE t (v VARIANT) or CAST(x AS VARIANT) cannot be planned. Spark 4.2 and DuckDB 1.5 both spell the type VARIANT in CREATE TABLE and CAST, so this is what lets a query written for them run unchanged.

• Register arrow.parquet.variant as a DFExtensionType (JSON display, and the identity the other hooks key on): the one unticked box in Implement DFExtensionType for Arrow's Canonical Extension Types #21144. Small; the natural first PR
• Map the SQL name VARIANT to that type via TypePlanner: Support Mapping SQL Types to the Canonical Extension Types #21626, same shape as UUID and JSON. The crate-side registration goes with it
• Pluggable cast resolution to and from extension types (CAST(x AS VARIANT), v::INT): Support pluggable cast resolution to support casting to and from extension types #20755, after the registration lands and a maintainer design. Not on our critical path

Adjacent, in arrow-rs

• The Parquet writer omits specification_version from the VARIANT logical type (LogicalType::variant(None)), so Spark 4.1 rejects the file with PARQUET_TYPE_NOT_RECOGNIZED. Reproduced 2026-09-25; unchanged on arrow-rs main as of 2026-10-19 (parquet/src/arrow/schema/extension.rs, logical_type_for_struct). One-line fix; issue to be filed on arrow-rs and linked here.

Describe alternatives you've considered

Keeping all of this inside OpenSearch (a fork of the Parquet reader and a private variant_get). Rejected: every hook above is generic to extension types, and the mixed-layout and statistics problems are hit by any Variant reader on DataFusion, not just ours.

Additional context

Compatibility target for the crate work is Spark 4.2.0 (datafusion-variant#69). We build against DataFusion 54 today and are moving to 55 / arrow-rs 59, which the crate requires.

Open questions for maintainers:

1. Are items 2 and 3 accepted as core concerns (neither mentions Variant), with the Variant adapter shipped from the crate?
2. For item 1, which shape: feat: expose UDF field access and input requirements for Parquet pruning #25013, feat: NestedInputAccess capability for nested read pruning and row group / page pruning #26282, or a merge of the two?
3. For item 4, ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307 or Add ExpressionAnalyzer for pluggable expression-level statistics estimation #21122?