feat: add Parquet content-defined chunking writer support - #3889
Open
kszucs wants to merge 3 commits into
Open
Conversation
PyArrow's ParquetWriter has supported content-defined chunking natively since 21.0.0, producing stable page boundaries across appends for content-addressable storage. Wire this through as write.parquet.content-defined-chunking.* table properties, mirroring the property names and defaults already used by iceberg-rust.
Raise a clear ValueError for invalid content-defined-chunking config (min-chunk-size <= 0, max-chunk-size <= min-chunk-size, negative norm-level) instead of letting PyArrow's opaque internal error surface. Also parametrize the near-duplicate kwargs tests, add coverage for the new validation, and make the write-path test assert use_content_defined_chunking actually reaches pq.ParquetWriter instead of only checking a round-trip that would pass even if the kwarg were silently dropped.
PyArrow already validates these values itself (e.g. max_chunk_size must be greater than min_chunk_size) and raises a clear OSError, and iceberg-rust's equivalent ParquetWriterBuilder::from_table_properties doesn't duplicate this validation either -- defer to PyArrow instead of maintaining a second, slightly different copy of the same checks.
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Adds support for PyArrow Parquet content-defined chunking (CDC) in the PyIceberg write path, exposing it via new table properties and guarding usage by the minimum supported PyArrow version.
Changes:
- Add new table properties for enabling/configuring Parquet CDC (enabled/min/max/norm-level).
- Wire CDC properties through
_get_parquet_writer_kwargs, including a shared_require_pyarrow_versionguard. - Add unit + integration-style tests and document the new properties.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| tests/io/test_pyarrow.py | Adds tests for CDC kwargs generation, version gating, and end-to-end wiring into pq.ParquetWriter. |
| pyiceberg/table/init.py | Introduces new CDC-related TableProperties constants and defaults. |
| pyiceberg/io/pyarrow.py | Adds _require_pyarrow_version helper, reuses it for Azure FS guard, and forwards CDC options to PyArrow writer kwargs. |
| mkdocs/docs/configuration.md | Documents new CDC table properties and their defaults/requirements. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+5524
to
+5525
| table_metadata = TableMetadataV2( | ||
| location=f"file://{tmp_path}", |
| } | ||
|
|
||
| assert len(data_files) == 1 | ||
| written_table = pq.read_table(data_files[0].file_path.replace("file://", "")) |
Comment on lines
+5512
to
+5514
| with pytest.raises(OSError, match="max_chunk_size must be greater than min_chunk_size"): | ||
| with pq.ParquetWriter(pa.BufferOutputStream(), table.schema, **kwargs) as writer: | ||
| writer.write_table(table) |
Comment on lines
+89
to
+90
| | `write.parquet.content-defined-chunking.min-chunk-size` | Size in bytes | 256KB | The minimum chunk size used for content-defined chunking | | ||
| | `write.parquet.content-defined-chunking.max-chunk-size` | Size in bytes | 1MB | The maximum chunk size used for content-defined chunking | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reopens #3608, which was closed by the stale bot. Rebased on current
main; GitHub refused to reopen the original PR after the branch was updated.Rationale for this change
PyArrow's
ParquetWriterhas natively supported content-defined chunking (CDC) since 21.0.0, producing stable page boundaries across appends (useful for content-addressable storage / dedup). PyIceberg's write path already funnels through a single kwargs builder (_get_parquet_writer_kwargs), so this wires CDC through aswrite.parquet.content-defined-chunking.*table properties, mirroring the property names and defaults iceberg-rust already uses for cross-engine consistency. Apyarrow>=21.0.0version guard raises a clearImportErrorif CDC is requested on an older PyArrow (extracted into a shared_require_pyarrow_versionhelper, reused by the existing Azure-filesystem version guard).Are these changes tested?
Yes: unit tests for
_get_parquet_writer_kwargs(disabled by default, enabled with defaults, enabled with custom values, unsupported PyArrow version) and an integration-style test that writes a table with CDC enabled end-to-end and reads it back.Are there any user-facing changes?
Yes: four new table properties (
write.parquet.content-defined-chunking.enabled,.min-chunk-size,.max-chunk-size,.norm-level), documented inconfiguration.md.