HIVE-29854: Iceberg: Estimate a pruned scan's column statistics from the table's manifests - #6740
Open
deniskuzZ wants to merge 2 commits into
Open
HIVE-29854: Iceberg: Estimate a pruned scan's column statistics from the table's manifests#6740deniskuzZ wants to merge 2 commits into
deniskuzZ wants to merge 2 commits into
Conversation
deniskuzZ
changed the title
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 30, 2026 10:22
d9600eb to
9013747
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 30, 2026 11:56
9013747 to
fee3e19
Compare
asf-ci-hive
added
tests unstable
tests pending
tests passed
and removed
tests pending
tests unstable
labels
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 31, 2026 14:25
fee3e19 to
dfcd4b0
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
August 31, 2026 18:26
dfcd4b0 to
b6abd65
Compare
asf-ci-hive
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 1, 2026 07:52
b6abd65 to
1c72550
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 1, 2026 09:19
1c72550 to
ae6dee8
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 1, 2026 20:48
115af67 to
11e1075
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 1, 2026 23:43
11e1075 to
a560fd8
Compare
asf-ci-hive
added
tests failed
tests pending
tests passed
and removed
tests pending
tests failed
labels
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 2, 2026 09:16
a560fd8 to
f24dc11
Compare
|
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 2, 2026 18:14
f24dc11 to
174b238
Compare
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
2 times, most recently
from
September 2, 2026 18:41
aafea8b to
0ca87bf
Compare
…ne them on read A partitioned table already kept its column statistics one entry per partition, but every gather held them all in memory, wrote each entry as a serialized Java object, and every read fetched one blob per partition and decoded all of it. None of that survives a table of any size. A gather is now pulled through the writer a batch at a time, so a table of two thousand partitions is never held at once, and an entry is written as the Thrift struct it is. A blob holds one partition, every column's entry behind the field it is for and its own length, so a scan decodes the columns it asked about and steps over the rest. The blobs a scan wants are read in as few requests as their places allow, merging what lies between them where crossing it costs less than the seek it saves - which the store says, not this. Every entry keeps the bit vector its distinct count is merged from. A count per partition is only worth having if the counts can be merged across the partitions a scan reads, and only a vector merges them: without one the fold falls back to the largest single partition's count, which for the partition column of an identity-partitioned table is one however many partitions are read. What still describes the data is decided from the snapshot a file was written for, so statistics separated from their rows are not served as though they stood. ANALYZE stores what it read, an overwrite what it replaced, and a major compaction of one current-spec partition refreshes it; a plain INSERT maintains none, and its partitions read as stale until recomputed. A branch's statistics stay on the branch, a file states every column it comes to hold so that whether a column has any is answerable from the table's metadata without opening it, and a file describing nothing is never committed in place of one that describes something. Where no stored statistics answer, the bounds the manifests already hold do, and they are folded a file at a time: a file states its metrics once for all of its columns, so what holds them is read where the file is in hand rather than again by each column of it. Where a column is one the spec partitions by identity, the partition is the value, so its bounds are exact and are stated even by a file that recorded no metrics of its own - which Iceberg stops inferring past its hundredth column.
…the table's manifests Column statistics stored for a table describe every partition of it, so a scan pruned to part of the table is handed bounds it mostly cannot hold. Where a column's values follow the partition key - a measure that grows with time, an id range loaded per batch - the bounds are wrong by the whole spread of the table, and a predicate outside the partition the scan reads is estimated as satisfiable. The manifests already record what each file holds. Where no per partition statistics answer, the bounds and null counts they state answer for the files the scan reads, folded a file at a time - a file states its metrics once for all of its columns, so what holds them is read where the file is in hand rather than again by each column of it. A column the spec partitions by identity is bounded by the partition value itself, which is exact and is stated even for a file that recorded no metrics of its own. They are not the answer, they are what the filter is then estimated from: a distinct count still comes from the statistics a gather stored, and a column without one states nothing. Planning the scan reads the manifests while a query compiles, so it plans with its own pool rather than the one split generation plans with. Off by default, under hive.iceberg.stats.use.manifests.
deniskuzZ
force-pushed
the
col_stats_manifest_scan
branch
from
September 2, 2026 23:01
0ca87bf to
bd4d154
Compare
asf-ci-hive
added
tests pending
tests failed
and removed
tests failed
tests pending
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
depends on #6707