Skip to content

Fix IbisSource IntegrityError on Snowflake with ibis >= 12 (filter before projecting categorical stats) - #282

Merged
cpsievert merged 1 commit into
mainfrom
fix/ibis-categorical-stats-integrity-error
Sep 1, 2026
Merged

Fix IbisSource IntegrityError on Snowflake with ibis >= 12 (filter before projecting categorical stats)#282
cpsievert merged 1 commit into
mainfrom
fix/ibis-categorical-stats-integrity-error

Conversation

@cpsievert

Copy link
Copy Markdown
Contributor

Problem

IbisSource.populate_column_stats() raises on Snowflake with
ibis-framework >= 12 when a table has 2+ low-cardinality text columns:

IntegrityError: Cannot add <NotNull ...> to filter, they belong to
another relation

The categorical-stats subqueries were built as:

table.select(...).filter(table[col].notnull())

The NotNull predicate belongs to the base table relation, but
.filter() is called on the projection returned by .select(). Ibis 12's
Filter.__init__ rejects predicates whose relations don't include the
filter's parent, and Snowflake's compile path constructs that invalid
Filter node directly.

Two details made this easy to miss:

  • It requires 2+ qualifying columnsibis.union() with one argument
    returns its input unchanged, so the union path that exposes the issue is
    never hit.
  • DuckDB doesn't reproduce it — its salvage rewrite rebinds the
    predicate and pushes the filter below the projection, masking the bug.

Fix

Filter before projecting, so the predicate belongs to the relation being
filtered:

table.filter(table[col].notnull()).select(...).distinct()

The subquery construction is extracted into
IbisSource._categorical_values_subquery() so the invariant is directly
testable. No behavior change otherwise.

Tests

  • Structural regression test: asserts the Filter node's parent is the
    base DatabaseTable (not a Project). This is the faithful repro —
    verified it fails on the old expression shape and passes on the new one.
    A behavioral DuckDB test alone cannot catch this bug.
  • Behavioral test: populate_column_stats populates categories for 2+
    low-cardinality text columns and excludes nulls.

Notes

  • Workaround for affected users until release: pass
    categorical_threshold=0 to QueryChat().
  • The R package is unaffected — it builds raw SQL strings via DBI rather
    than ibis expression graphs.

The categorical-stats subquery applied .filter() after .select(),
leaving a NotNull predicate bound to the base table on a Filter whose
parent is the projection. Snowflake (ibis >= 12) rejects this with
IntegrityError. Filter before projecting instead.
@cpsievert
cpsievert force-pushed the fix/ibis-categorical-stats-integrity-error branch from b256efd to 2e1a26e Compare August 31, 2026 21:45
@cpsievert
cpsievert merged commit 39bbdf5 into main Sep 1, 2026
7 checks passed
@cpsievert
cpsievert deleted the fix/ibis-categorical-stats-integrity-error branch September 1, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant