Search before asking
Motivation
DataFusion queries that group by partition columns and calculate COUNT(*) currently plan every live file and read data files. On tables with very large file counts, planning full manifest entries and column statistics can exhaust memory even though the result is derivable from metadata.
Data-evolution tables also cannot simply sum per-file row counts because multiple column-group or blob files may cover the same row-ID ranges, and deletion-vector rows must be excluded.
Solution
Add an exact manifest-only path for COUNT(*) grouped by partition columns:
- stream only the required manifest fields and apply partition pruning;
- net ADD/DELETE entries using complete file identity;
- union row-ID ranges only when data evolution is enabled;
- subtract known deletion-vector cardinalities;
- fall back to the ordinary scan whenever metadata cannot provide an exact answer.
The implementation should avoid retaining all live file metadata and statistics. Its aggregation state may still grow with live DELETE entries and fragmented, non-adjacent row-ID ranges.
Anything else?
Implementation: #858
Willingness to contribute
Search before asking
Motivation
DataFusion queries that group by partition columns and calculate
COUNT(*)currently plan every live file and read data files. On tables with very large file counts, planning full manifest entries and column statistics can exhaust memory even though the result is derivable from metadata.Data-evolution tables also cannot simply sum per-file row counts because multiple column-group or blob files may cover the same row-ID ranges, and deletion-vector rows must be excluded.
Solution
Add an exact manifest-only path for
COUNT(*)grouped by partition columns:The implementation should avoid retaining all live file metadata and statistics. Its aggregation state may still grow with live DELETE entries and fragmented, non-adjacent row-ID ranges.
Anything else?
Implementation: #858
Willingness to contribute