Skip to content

feat(spec): read partition and sort fields with multi-argument transforms (source-ids)#2802

Open
moomindani wants to merge 2 commits into
apache:mainfrom
moomindani:multi-arg-source-ids-read
Open

feat(spec): read partition and sort fields with multi-argument transforms (source-ids)#2802
moomindani wants to merge 2 commits into
apache:mainfrom
moomindani:multi-arg-source-ids-read

Conversation

@moomindani

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

The v3 spec serializes partition fields and sort fields whose transform takes multiple arguments with a source-ids list instead of the single source-id. Deserializing such metadata failed with missing field source-id, so a v3 table containing any multi-argument transform could not be read at all.

This adds read tolerance, mirroring PyIceberg (apache/iceberg-python#3630):

  • PartitionField and SortField accept source-ids: a single-element list is normalized onto source-id; a multi-element list is kept in the new source_ids field and the transform maps to Transform::Unknown — multi-argument transforms cannot be evaluated, and the spec requires v3 readers to read such tables ignoring them. source_id holds the first id so existing consumers keep working.
  • Serialization follows the spec's writer rule: single-argument transforms write only source-id, multi-argument transforms write only source-ids, so the field round-trips.
  • PartitionSpec::is_compatible_with also compares source_ids, so two multi-arg fields sharing the first id no longer compare as identical.
  • SortField's Display renders all ids for multi-argument fields.
  • PartitionSpec::partition_type works unchanged: Transform::Unknown produces string per the spec.

Note: this adds a public source_ids field to PartitionField and SortField (will be flagged by cargo-public-api). Known limitation shared with #2790: Transform::Unknown does not retain the original transform name, so round-tripping writes "unknown"; name preservation stays tracked in #2789.

Are these changes tested?

Yes — new unit tests in spec/partition.rs and spec/sort.rs: multi-arg deserialization and round-trip, single-element normalization, empty source-ids rejection, missing-transform rejection, and partition_type over a multi-arg field. cargo test -p iceberg --lib (1399 tests), cargo clippy -p iceberg --lib --tests, and cargo check --workspace --tests pass locally.

This pull request and its description were written by Claude Fable 5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reading v3 metadata with multi-argument transform fields (source-ids) fails

1 participant