Skip to content

[Feature] Push down partitioned COUNT(*) to manifest metadata #859

Description

@jerry-024

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

DataFusion queries that group by partition columns and calculate COUNT(*) currently plan every live file and read data files. On tables with very large file counts, planning full manifest entries and column statistics can exhaust memory even though the result is derivable from metadata.

Data-evolution tables also cannot simply sum per-file row counts because multiple column-group or blob files may cover the same row-ID ranges, and deletion-vector rows must be excluded.

Solution

Add an exact manifest-only path for COUNT(*) grouped by partition columns:

  • stream only the required manifest fields and apply partition pruning;
  • net ADD/DELETE entries using complete file identity;
  • union row-ID ranges only when data evolution is enabled;
  • subtract known deletion-vector cardinalities;
  • fall back to the ordinary scan whenever metadata cannot provide an exact answer.

The implementation should avoid retaining all live file metadata and statistics. Its aggregation state may still grow with live DELETE entries and fragmented, non-adjacent row-ID ranges.

Anything else?

Implementation: #858

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions