Repository navigation
perf(compile): stop repeating per-call SQL macro work - #1549
Merged
Merged
Conversation
A model with many SQL macro calls re-did per-macro work on every call: the returns check rebuilt its probe, declared types were normalized, the macro file was parsed and the caller's text was lexed again, and the authored file was searched for the query. Each now happens once per macro or per pass. SQL-test patterns compile once per process.
Turning SQL analysis off skips type inference but not column lineage; --lineage-mode none turns lineage off.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Models with many SQL macro calls spent most of compile on the main thread repeating per-macro work for every call. On 1,000 models with 40 typed calls each, a profile put 32% of samples in macro expansion and 16% more in re-normalising the same declared types.
Changes
returnscheck memoises its verdict by dialect, adapter, definition and body, so each macro is probed once, not once per call.expand_sql_macro_calls); usage facts are still recorded per call.--lineage-mode none. Behaviour is unchanged.Outputs, diagnostics and locations are unchanged. Memos live for the process, like the existing body-type memos. Still deferred without profile support: pool size, the Snowflake publish step, the second
share_keynormalisation,Arcshapes and probe-name caching.No DuckShop case: no new project shape, value or lifecycle; DuckShop output is identical.
Verification
returnsdeclarations is judged on each; it fails if the memo ignores the definition.make test: all pass except Docker-worker publication cases needing a local image (as on main);make check-cipasses on this head.