fead: add Parquet content-defined chunking writer support - #3608
Conversation
PyArrow's ParquetWriter has supported content-defined chunking natively since 21.0.0, producing stable page boundaries across appends for content-addressable storage. Wire this through as write.parquet.content-defined-chunking.* table properties, mirroring the property names and defaults already used by iceberg-rust.
kszucs
changed the title
Raise a clear ValueError for invalid content-defined-chunking config (min-chunk-size <= 0, max-chunk-size <= min-chunk-size, negative norm-level) instead of letting PyArrow's opaque internal error surface. Also parametrize the near-duplicate kwargs tests, add coverage for the new validation, and make the write-path test assert use_content_defined_chunking actually reaches pq.ParquetWriter instead of only checking a round-trip that would pass even if the kwarg were silently dropped.
PyArrow already validates these values itself (e.g. max_chunk_size must be greater than min_chunk_size) and raises a clear OSError, and iceberg-rust's equivalent ParquetWriterBuilder::from_table_properties doesn't duplicate this validation either -- defer to PyArrow instead of maintaining a second, slightly different copy of the same checks.
2065254 to
466ca6a
Compare
|
@anxkhn could you please take a look? |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
Reopened as #3889 (rebased on current main) — GitHub wouldn't let this one be reopened after the branch was updated. |
1 participant
Rationale for this change
PyArrow's
ParquetWriterhas natively supported content-defined chunking (CDC) since 21.0.0, producing stable page boundaries across appends (useful for content-addressable storage / dedup). PyIceberg's write path already funnels through a single kwargs builder (_get_parquet_writer_kwargs), so this wires CDC through aswrite.parquet.content-defined-chunking.*table properties, mirroring the property names and defaults iceberg-rust already uses for cross-engine consistency. Apyarrow>=21.0.0version guard raises a clearImportErrorif CDC is requested on an older PyArrow (extracted into a shared_require_pyarrow_versionhelper, reused by the existing Azure-filesystem version guard).Are these changes tested?
Yes: unit tests for
_get_parquet_writer_kwargs(disabled by default, enabled with defaults, enabled with custom values, unsupported PyArrow version) and an integration-style test that writes a table with CDC enabled end-to-end and reads it back.Are there any user-facing changes?
Yes: four new table properties (
write.parquet.content-defined-chunking.enabled,.min-chunk-size,.max-chunk-size,.norm-level), documented inconfiguration.md.