Skip to content

feat: built-in Snowflake performance rules for JSON copied onto finer rows - #1515

Merged
chio-labs-delivery[bot] merged 21 commits into
mainfrom
feat/performance-rules
Nov 16, 2026
Merged

chio-labs-delivery[bot] merged 21 commits into
mainfrom
feat/performance-rules

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Why

Two Snowflake checks for JSON duplicated across rows were only custom Python rules.

Changes

  • New SQBRPERFORMANCE family, Snowflake only, off by default; select each rule by code.
  • SQBRPERFORMANCE101 flags an output column copying a FLATTEN value whole when a deeper FLATTEN in the same query reads its child array.
  • SQBRPERFORMANCE102 flags a semi-structured output column read from an input with a coarser declared unique key. Exempt: models read only by published views of their entity.
  • Models without lineage or types are skipped and counted in a note (--verbose lists them).
  • On Snowflake, selecting 102 makes every edit a full compile (dense 3,000-model edit: 3.3–4.0× wall); documented. Delta support is a follow-up.
  • All-rules benchmarks select 101 and 102; edit benchmarks don't.
  • Bindings cap 197,891 → 197,963 (+77 with sqlbuild-python-base) for 102's lineage, keys and note.

Ticket interpretations:

  • 101: a copy is o.value itself, through casts, conversions or OBJECT_CONSTRUCT/OBJECT_INSERT/ARRAY_CONSTRUCT. Paths, other functions and IFF are not. OBJECT_DELETE passes only when it removes the key the deeper FLATTEN reads.
  • 102: semi-structured means inferred VARIANT, OBJECT or ARRAY. JSON in VARCHAR is not detected.
  • 102: keys are declared (non-null unique columns, merge unique_key, one-row seeds). Unkeyed models and aggregates are not checked; a model key covered by an input key exempts it. Every reader must be such a view.

Verification

  • Rust cases for both rules; an integration test compiles the fixture; a delta test covers the Snowflake-only fallback.
  • Snowflake trial: a copied payload is VARIANT, repeated per line.
  • Dense 3,000 cold, off vs on, same binary, 3 runs: median wall 7.22/7.23 s, CPU 22.15/22.40 s.
  • make check-ci passes. Large-project comparison with the reference rules remains open.

DuckShop: new edge_cases/performance_rules fixture.

@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 16, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 16, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 16, 2026
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 16, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 16, 2026
SQBRPERFORMANCE102 compares models across the project, so a Snowflake
compile that selects it never takes the delta path. Selecting it in the
shared dense project made the delta edit loop and the base/head edit
ratio measure full compiles. The default dense selection is back to the
one main uses, and only the all-rules cold benchmark and the varied
benchmark select both performance rules.
Selecting the rule on the Snowflake dialect makes every edit compile the
whole project. The compile page lists it among the full-compile cases and
the performance rules page says so where it explains selecting the rules.
A delta compile test covers both sides: the rules selected on DuckDB keep
the delta, and on Snowflake the fallback names SQBRPERFORMANCE102.
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit f450b25 Nov 16, 2026
36 checks passed
@chio-labs-delivery
chio-labs-delivery Bot deleted the feat/performance-rules branch November 16, 2026 12:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant