Repository navigation
[C++][Parquet][Python] New API to 'zip' or (vertically) 'attach' parquet metadata #40958
Closed as not planned
Description
Describe the enhancement requested
Hi,
One of the design principles of parquet from their Github page is 'Separating metadata and column data':
Separating metadata and column data.
The format is explicitly designed to separate the metadata from the data. This allows splitting columns into multiple files, as well as having a single metadata file reference multiple parquet files.
In order to achieve the 'columns in different files', we need to
- Ensure each file has the same number of row-groups
- Ensure each corresponding row-group of each file have the same rows
- Grab the 'metadata' from each file, 'zip/attach them vertically', and write out the new metadata file
- Feed this metadata while reading the table
It looks like Arrow APIs provide nearly everything to achieve this, except for the bolded portion in point 3 above.
This ticket is requesting the addition of a new API to be able to 'zip'/'join'/'attach' metadata from 2 files.
For e.g.
import pyarrow.parquet as pq
m1 = pq.read_metadata('file1.parquet') # say this has columns: col1, col2, col3
m1.set_file_path('file1.parquet')
m2 = pq.read_metadata('file2.parquet') # say this has columns: col4, col5
m2.set_file_path('file2.parquet')
# requesting this new 'zip' API
m = m1.zip(m2) # needs to ensure same number of row groups, and same number of rows within each row group
# m will now have metadata for col1, col2, col3, col4, col5 each pointing to appropriate data file
m.write_metadata('_metadata')One this is done, a combined data can be created using:
m = pq.read_metadata('_metadata')
data = pq.ParquetFile('file1.parquet', 'file2.parquet', metadata=m)
# data should now be able to show all columnsComponent(s)
C++, Python, Parquet
Activity
[-]API to 'zip' or (vertically) 'attach' parquet metadata[/-][+]New API to 'zip' or (vertically) 'attach' parquet metadata[/+][-]New API to 'zip' or (vertically) 'attach' parquet metadata[/-][+][C++][Parquet][Python] New API to 'zip' or (vertically) 'attach' parquet metadata[/+]mapleFU commentedon Apr 9, 2024
@wgtmac does spec allowing this currently?
wgtmac commentedon Apr 10, 2024
I think this is the parquet summary metadata file. See
parquet.summary.metadata.levelfrom https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/README.md#class-parquetoutputformat. But I don't know whether it is widely used.mrbrahman commentedon Apr 11, 2024
@wgtmac, no I don't think _metadata file would be widely used in the bigdata systems like Hadoop/Spark etc. However, with Apache Arrow it does seem to have the required API (in ParquetFile) to read metadata separately from the data.
Of course, I'm also not sure if Apache Arrow will also specifically support the below section from my request (because we currently have no way to stitch 2 metadata files):
where file1.parquet contains col1, col2, co3 and file2.parquet contains col4 and co5 (different set of columns). Here only the the _metadata file has the overarching information about the 'table' definition.
I'm only guessing that it would be supported, since it has the API to do so. However, it would be nice to confirm that as well.
wgtmac commentedon Apr 12, 2024
Before talking about the
zipAPI, it still need some refactoring work to support the metadata file. What is your use case then? The possible use case for metadata file is to combine parquet files of different columns into a larger logical parquet file. From my perspective, if metadata file is not widely used, it seems not worth the effort to implement it.mrbrahman commentedon Apr 13, 2024
@wgtmac, sorry I didn't quite understand
pyarrow already supports metadata file, right?
https://arrow.apache.org/docs/python/parquet.html#writing-metadata-and-common-metadata-files
wgtmac commentedon Apr 13, 2024
Awkward. I even didn't notice that this is already supported. Thanks for pointing it out!
mrbrahman commentedon Jun 5, 2024
@wgtmac, any further insights into this.
I think the only place that would need to change is https://github.com/apache/arrow/blob/main/cpp/src/parquet/metadata.cc
wgtmac commentedon Jun 13, 2024
@mrbrahman This feature is controversial and recently there is a related discussion: apache/parquet-format#242 (comment).
github-actions commentedon Nov 18, 2025
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
thisisnic commentedon Feb 18, 2026
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.