[WIP] HIVE-29737: Prune Parquet row groups with bloom filters in the vectorized reader - #6741
Open
deniskuzZ wants to merge 1 commit into
Open
[WIP] HIVE-29737: Prune Parquet row groups with bloom filters in the vectorized reader#6741deniskuzZ wants to merge 1 commit into
deniskuzZ wants to merge 1 commit into
Conversation
Contributor
|
I did |
deniskuzZ
force-pushed
the
HIVE-29737
branch
6 times, most recently
from
August 30, 2026 20:33
522a484 to
02b7a03
Compare
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
deniskuzZ
changed the title
deniskuzZ
changed the title
asf-ci-hive
added
tests passed
tests pending
and removed
tests pending
tests passed
labels
Member
Author
|
depends on |
Contributor
yes in https://iceberg.apache.org/docs/nightly/configuration/#write-properties nightly docs they have exposed it. |
Member
Author
that's always confusing with iceberg what is in spec and what's actually usable/supported by library. read.split.adaptive-size.enabled is not supported even in latest release |
deniskuzZ
force-pushed
the
HIVE-29737
branch
from
September 1, 2026 11:33
40974ee to
0ab5653
Compare
deniskuzZ
force-pushed
the
HIVE-29737
branch
2 times, most recently
from
September 1, 2026 13:07
9e13d57 to
cde8af4
Compare
deniskuzZ
force-pushed
the
HIVE-29737
branch
2 times, most recently
from
September 1, 2026 13:48
0949b3b to
f3d865f
Compare
…ized reader Parquet files can carry bloom filters, and Iceberg tables ask for them through the write.parquet.bloom-filter-enabled.column.* table properties, but the vectorized reader never consulted them. VectorizedParquetRecordReader builds its reader from the original job conf, where ParquetInputFormat.getFilter yields a NoOpFilter. Row groups are therefore pruned only by ParquetRecordReaderBase#getSplit, which asked RowGroupFilter for statistics. A value inside a column's min/max range but absent from the data was never skipped. The non-vectorized reader was unaffected: it builds its context from the conf that carries the pushed down predicate, so parquet applies all filter levels there already. This sits under MapredParquetInputFormat's vectorized branch as much as under Iceberg's reader, so it applies to any Parquet table read vectorized, whoever wrote the file. Run the bloom filter level over the row groups that survive statistics. Bloom filters live in the data file rather than the footer, so this needs an open reader, and it is opened only when a surviving row group carries a filter for a column the predicate can prune on. A bloom filter only proves a value absent, so that is the columns under equality and set membership, which is what BloomFilterImpl reads; an OR contributes only when both of its sides do. The reader is built without a record filter, since parquet would otherwise repeat the whole filtering pass in the constructor. Parquet's own parquet.filter.bloom.enabled turns the level off. Under LLAP the reader otherwise reaches past the cache for those bytes: only the footer was cached, so every reader re-read the filters from the file. The column data cache cannot hold them either, as it is indexed by column chunk while bloom filters sit outside every chunk, and its reads must start at a registered chunk. Cache them the way ORC caches the metadata its bloom indexes ride in. MetadataCache gains get/putParquetBloomFilters, keyed by file and by the offset the footer records for the column chunk. Keying per filter rather than per file keeps an entry describing exactly the bytes it holds, and keeps its size to one filter: parquet sizes these from its own defaults, roughly a megabyte per column per row group, so a file wide entry would pull every row group's filters on the first miss into a cache that otherwise holds footers of a few hundred bytes. A reader asks for the filters of its own row groups in one call, which opens the file once for whatever is missing, and presents them to parquet through ParquetFilterDataFromCache, a sparse InputFile serving cached ranges at the offsets the file stores them at. The buffers stay locked until the filtering pass is done, as the cache is what the allocations for the remaining filters evict from. Without LLAP, or when a column chunk records no bloom filter length, the plain reader reads the file as before. TestParquetRowGroupFilter covers the pruning rules over plain Parquet files, including the predicate shapes that must not open the file, and TestParquetFilterDataFromCache covers serving several cached ranges at their own offsets. TestMetadataCache covers evicting one filter while the file's footer and its other filters stay cached. iceberg_parquet_bloom_filter.q covers the write path and end to end pruning, and llap_iceberg_bloom_filter.q covers the LLAP path over both a single row group and a file of several, where hive.llap.io.cache.only makes the repeated queries fail unless the filters come from the cache.
deniskuzZ
force-pushed
the
HIVE-29737
branch
from
September 1, 2026 14:16
f3d865f to
7ab169e
Compare
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Run the bloom filter level over the row groups that survive statistics.
Why are the changes needed?
Iceberg tables can write Parquet bloom filters through the write.parquet.bloom-filter-enabled.column.* table properties, and Hive inserts honor them, but the vectorized reader never consulted them.
Does this PR introduce any user-facing change?
No
How was this patch tested?