Skip to content

fix(query): preserve empty-input distinct counts - #65

Merged
prrao87 merged 1 commit into
mainfrom
codex/fix-aggregate-predicates
Oct 27, 2026
Merged

prrao87 merged 1 commit into
mainfrom
codex/fix-aggregate-predicates

Conversation

@stugorf

@stugorf stugorf commented Oct 27, 2026

Copy link
Copy Markdown
Member

Problem and change

MATCH (p:Person) WHERE p.id = 99 RETURN count(DISTINCT p.id) > 0 AS present can return true when no row matches, because DataFusion's statistics shortcut turns the empty distinct count into 1.

Construct a query-local frame that omits aggregate_statistics only when the logical plan contains a distinct count. Preserve the caller session, the other optimizer rules, its query planner and the deadline context. Execution, streaming and explain share the guard. Current main already supports aggregates inside comparison/CASE expressions; this PR does not duplicate that work.

Refs lance-format/lance-graph#123. This is a lance-csr fix; it does not close the upstream issue.

Validation

  • Failing before on the verified 0.3.1 debug baseline: 30 failed / 94 passed. Passing after: 124 focused cases, 1,387 complete Python tests in both debug and isolated production-wheel verification.
  • Passed: cargo fmt --all --check, cargo test --workspace --locked, workspace Clippy with warnings denied, make install, make test, make lint, executable documentation snippets, strict Zensical build and typos@1.26.0.
  • All three candidate debug gates passed before any final production build. Fresh locked consumers outside the checkouts verified wheel/package payloads and native hashes, then ran each complete suite with Rust absent from PATH.

Empty and nonempty counts, booleans, CASE, WITH, duplicates, nulls, parameter reuse, output types, caller-session preservation, legacy/join/CSR execution and streaming over Arrow/Lance.

Workload Mode Correct queries First-pass geometric speedup
30-query SF1 memory 30 0.924x
30-query SF1 lazy 30 1.024x
IC SF1 memory 14 0.994x
IC SF1 lazy 14 0.999x

Above 1 is faster. These original suite means are retained unchanged; initial >10% per-query flags and their paired investigations are recorded in the final gate record. Both complete workloads preserve the original golden answers. These are local engineering measurements, not an audited SNB result or performance guarantee.

Measured serially after builds stopped, on Apple M4 Max (16 cores, 128 GiB), macOS arm64, Rust 1.96.0 and Python 3.13.17. The suites are pinned at 6260689 (30 queries) and c185123 (IC); settings, dependencies, SF1 manifests, timings and peak process RSS are recorded. Per-query before/after tables, reproduction instructions, and hashed raw evidence provide the commands and receipts.

User impact

Correct empty-input distinct counts and derived predicates. The Cypher guide documents the count semantics and scoped optimizer guard. No Python signatures, stubs or dependencies change.

Independently based on f5a986e822032a3aa8c52af55ad08e92504ee73c, head 80d4438884872ce9d18c528caa1519441a9460ce. Latest main 6effd0db0c002912cf969fd7447ea7a3ce2631df adds documentation links/package URLs only; runtime sources and locks are unchanged, and this PR applies cleanly. Other platforms and hosted CI are not included in the local verification; check the PR checks for their current status.

Submitted by David Hughes (@stugorf), with AI Collaboration.

Refs lance-format/lance-graph#123. Adapt the focused fix to the current 0.3.1 engine and crate layout.

Submitted by David Hughes (@stugorf), with AI Collaboration.
@prrao87
prrao87 merged commit dbb8e9f into main Oct 27, 2026
8 checks passed
@prrao87
prrao87 deleted the codex/fix-aggregate-predicates branch October 27, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants