Skip to content

feat(writer): apply per-column parquet bloom filter write properties - #3148

Open
alessandro-nori wants to merge 2 commits into
apache:mainfrom
alessandro-nori:anori/parquet-bloom-filter-write-properties
Open

feat(writer): apply per-column parquet bloom filter write properties#3148
alessandro-nori wants to merge 2 commits into
apache:mainfrom
alessandro-nori:anori/parquet-bloom-filter-write-properties

Conversation

@alessandro-nori

@alessandro-nori alessandro-nori commented Sep 4, 2026

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

The Parquet writer ignored write.parquet.bloom-filter-enabled.column.*, -fpp.column.* and -ndv.column.*, so a table configured for bloom filters silently got files without them.

  • TableProperties: three prefixed-map properties for the per-column bloom settings.
  • ParquetWriterBuilder::from_table_properties: applies them to the parquet-rs writer properties. Column names resolve to Parquet leaf paths through the existing IndexByParquetPathName visitor, so list elements and map keys work (tags.elementtags.list.element).
  • Columns missing from the schema are logged and skipped; an out-of-range fpp or a zero ndv is rejected with DataInvalid.

Tables already carrying these properties will start producing bloom filters.

Differences from Iceberg Java:

  • bloom-filter-max-bytes and bloom-filter-adaptive-enabled have no parquet-rs equivalent and are not implemented.
  • An enabled column gets fpp 0.01 even with no -fpp.column. set: parquet-rs defaults to 0.05, parquet-mr to 0.01.
  • fpp/ndv are skipped when -enabled.column. is false, since both setters implicitly re-enable the filter.
  • A malformed value fails the write rather than being warned past, consistent with the other TableProperties.

Are these changes tested?

Unit tests cover enable/disable, fpp/ndv application, nested struct columns, list elements and map keys, unknown and non-leaf columns, and both validation errors. An end-to-end test writes a file and reads the filter back, asserting a written value probes positive, an absent one negative, and an unconfigured column carries no filter.

AI Disclosure

Developed with AI assistance (Claude Code) for implementation, tests and review. I have reviewed and verified the change and take responsibility for its content.

@alessandro-nori
alessandro-nori force-pushed the anori/parquet-bloom-filter-write-properties branch 4 times, most recently from fc1c3d1 to 8033af8 Compare September 4, 2026 14:16
@alessandro-nori
alessandro-nori force-pushed the anori/parquet-bloom-filter-write-properties branch from 8033af8 to 9bc3cf9 Compare September 4, 2026 14:33
@alessandro-nori alessandro-nori changed the title feat: apply per-column parquet bloom filter write properties feat(writer): apply per-column parquet bloom filter write properties Sep 4, 2026
@alessandro-nori
alessandro-nori marked this pull request as ready for review September 4, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet writer ignores per-column write.parquet.bloom-filter-* table properties

1 participant