Repository navigation
Conversation
Switch the default Parquet data page version from V1 to V2 across C++ and Python (PyArrow). V2 data pages enable per-page compression selectivity (via min_compression_savings) and include explicit page-level null count metadata (num_nulls) in the page header. Closes apache#55272.
Awesome06
requested review from
AlenkaF,
HuaHuaY,
pitrou,
raulcd,
rok and
wgtmac
as code owners
November 9, 2026 12:38
|
|
… masking and synchronize V2 page defaults
Reorder MaybeReplaceValidity before update_stats in WriteArrowDictionary so writeable_indices reflects parent nulls prior to computing page stats, and guard dictionary reuse with chunk_indices->null_count() == 0.
Update TestSerialize for BooleanType in Parquet C++ to account for V2 selective compression keeping incompressible 7-byte boolean pages uncompressed.
Update test_parquet_metadata_api expected boolean encodings to {'RLE'} for V2 defaults.
Synchronize parquet.rst Sphinx doctest expected row group byte size for V2 headers.
…ion in TestSerialize and synchronize Sphinx doctest sizes Update TestSerialize in Parquet C++ to allow selective compression leaving small or incompressible pages uncompressed under V2 page defaults. Wrap comment to adhere to 90-char limit for clang-format. Synchronize column chunk total_compressed_size and total_uncompressed_size in parquet.rst Sphinx doctest for V2 page defaults.
…V2 data pages Update TestSerialize in Parquet C++ to check positivity of compressed and uncompressed sizes under V2 data pages rather than asserting strict ordering. Under V2, selective compression can leave data pages uncompressed, while unconditional compression of dictionary pages can expand small payloads, making total_compressed_size less than, equal to, or greater than total_byte_size.
adamreeve
reviewed
Nov 11, 2026
adamreeve
left a comment
adamreeve
left a comment
Contributor
There was a problem hiding this comment.
Thanks for this @Awesome06! I have a couple of questions
…atistics with nulls Add TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength to verify unreferenced dictionary values are not included in statistics when null count causes unique count to equal dictionary length. Add TestArrowWriteDictionary.NestedDictionaryParentNullStatistics to verify that nested dictionary columns under nullable parents accurately reflect ancestor nulls in page and chunk statistics across both V1 and V2 data pages.
Awesome06
force-pushed
the
gh-55272-parquet-default-v2-pages
branch
from
November 11, 2026 02:45
d61267d to
fd23951
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Parquet V2 data pages are widely supported across the ecosystem. They provide explicit page-level null counts (
num_nulls) in page headers and enable per-page selective compression (min_compression_savings), allowing pages that do not benefit from compression to remain uncompressed.Benchmarks across TPC-H, SSB, and TPC-DS (discussed on GH-55272) showed that V2 data pages yield comparable file sizes (0.0% to 1.2% delta depending on schema sparsity) while improving read throughput by 12% to 35% on scanned columns.
What changes are included in this PR?
properties.h):data_page_versioninWriterProperties::BuilderfromParquetDataPageVersion::V1toParquetDataPageVersion::V2.V2default.properties_test.cc,column_writer_test.cc):TestWriterProperties.Basicsto expectParquetDataPageVersion::V2.MinCompressionSavingsRequiresDataPageV2to verify defaultV2allowsmin_compression_savingswithout throwing, while explicitly configuredV1raises an exception.WriteLargeDictEncodedPage, updated expected page type toPageType::DATA_PAGE_V2.parquet.rst):data_page_versiondefaults to V2, with V1 retained for older reader compatibility.core.py,_dataset_parquet.pyx,_parquet.pyx):data_page_versionfrom'1.0'to'2.0'inParquetWriter,write_table, and dataset write options._parquet.pyxvalidation somin_compression_savings > 0.01only rejects explicitdata_page_version="1.0".test_parquet_writer.py):test_min_compression_savings_rejects_v1to explicitly specifydata_page_version="1.0".test_min_compression_savings_default_v2to verify default write paths acceptmin_compression_savingswithout error.Are these changes tested?
Yes. Updated existing C++ properties and writer unit tests, and added Python test coverage for default V2 writer behavior.
Are there any user-facing changes?
Yes. Files written with default writer properties in C++ and PyArrow will now use Parquet Data Page V2 instead of V1. Users requiring V1 for legacy compatibility can pass
data_page_version="1.0"(Python) or.data_page_version(ParquetDataPageVersion::V1)(C++).Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: