Skip to content

Shrink plans for comprehension-heavy queries such as IC10 #18

Open

Description

@prrao87

Problem

Queries built from pattern comprehensions, OPTIONAL MATCH, collect() and chained WITH produce very large physical plans. Two of them, IC10 and IC1, are where lance-csr trails other engines most; IC14 has the same plan growth, although it is already faster than the other engines.

Evidence

Physical operator counts on SF1, from explain():

Query Operators Notable lance-csr in memory Fastest other engine
IC10 217 38 AggregateExec, 32 CsrExpandExec 105.0 ms 13.7 ms (Ladybug)
IC14 225 49 RepartitionExec 64.4 ms 522.6 ms (Neo4j)
IC1 164 20 AggregateExec 26.8 ms 13.5 ms (Kuzu)

IC10 collects each friend's posts and counts those that share a tag with the person, with size([p IN posts WHERE (p)-[:postHasTag]->()<-[:hasInterest]-(person)]). An earlier change (seeding comprehensions from the original rows) cut IC14 from 641 to 225 operators and from 188 to 62 ms, so there is precedent for large wins here.

Suggested fix

Start with IC10: read its logical plan and find where the comprehension and the OPTIONAL MATCH ... collect() are planned as separate joins and aggregations over the same rows. Candidates are evaluating a pattern predicate inside a list comprehension as a semi-join over the unnested list, and merging aggregations that group by the same keys.

Acceptance criteria

  • IC10's operator count and latency before and after, in the pull request, with results unchanged.
  • No regression on the other IC queries or the 30-query workload.
  • Any new planning rule has focused tests.

Identified by profiling the LDBC IC benchmark; IC10's plan has not been analysed in detail.

Activity

  1. prrao87 commented on Oct 29, 2026

    @prrao87
    MemberAuthor

    Partial progress from #38, which counts size([x IN list WHERE ...]) without building the list. On current main (Apple M5, SF1, in memory), IC10 is down from 217 to 201 operators (AggregateExec 38 → 32, CsrExpandExec 32 → 24) and from 105.0 to 97.8 ms, still about 7x slower than Ladybug's 13.7 ms. IC1 is unchanged at 164 operators. The suggested fixes, a semi-join for the pattern predicate and merging aggregations with the same keys, are still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions