Skip to content

perf(compile): stop repeating per-call SQL macro work - #1549

Merged
kvlonge merged 2 commits into
mainfrom
perf/compile-hotspots
Nov 17, 2026
Merged

kvlonge merged 2 commits into
mainfrom
perf/compile-hotspots

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Why

Models with many SQL macro calls spent most of compile on the main thread repeating per-macro work for every call. On 1,000 models with 40 typed calls each, a profile put 32% of samples in macro expansion and 16% more in re-normalising the same declared types.

Changes

  • The returns check memoises its verdict by dialect, adapter, definition and body, so each macro is probed once, not once per call.
  • Declared type families are memoised per thread.
  • A declaration pass parses each macro file once and lexes the caller's text once for all its calls (expand_sql_macro_calls); usage facts are still recorded per call.
  • A pass finds where the authored query starts once, not per recorded call.
  • SQL-test patterns compile once per process instead of per scenario and test.
  • Docs: turning SQL analysis off does not turn off lineage. The semantic compilation page said it did; it now points to --lineage-mode none. Behaviour is unchanged.

Outputs, diagnostics and locations are unchanged. Memos live for the process, like the existing body-type memos. Still deferred without profile support: pool size, the Snowflake publish step, the second share_key normalisation, Arc shapes and probe-name caching.

No DuckShop case: no new project shape, value or lifecycle; DuckShop output is identical.

Verification

  • Typed project (1,000 models × 40 calls), 5 alternating runs, same machine, load 1.4–3.2 on 8 cores: wall 12.14→7.35 s, CPU 15.29→10.07 s, RSS unchanged; JSON (2,000 diagnostics) and files identical.
  • DuckShop, 3 runs: wall 2.21→2.11 s, CPU 6.46→6.13 s; identical.
  • Dense 3,000 models, 7 runs: wall 7.80→7.65 s, CPU 24.72→24.15 s; identical.
  • New test: one body under two returns declarations is judged on each; it fails if the memo ignores the definition.
  • 334 SQL macro E2E/integration tests pass.
  • make test: all pass except Docker-worker publication cases needing a local image (as on main); make check-ci passes on this head.

A model with many SQL macro calls re-did per-macro work on every call:
the returns check rebuilt its probe, declared types were normalized,
the macro file was parsed and the caller's text was lexed again, and
the authored file was searched for the query. Each now happens once per
macro or per pass. SQL-test patterns compile once per process.
Turning SQL analysis off skips type inference but not column lineage;
--lineage-mode none turns lineage off.
@chio-labs-delivery
chio-labs-delivery Bot added this pull request to the merge queue Nov 17, 2026
@kvlonge
kvlonge removed this pull request from the merge queue due to a manual request Nov 17, 2026
@kvlonge
kvlonge added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 4740e84 Nov 17, 2026
34 checks passed
@kvlonge
kvlonge deleted the perf/compile-hotspots branch November 17, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant