Skip to content

perf(analysis): check SQL-macro arguments on the analysis pool and drop duplicate work - #1506

Merged
kvlonge merged 1 commit into
mainfrom
perf/parallel-sql-macro-probes
Nov 15, 2026
Merged

kvlonge merged 1 commit into
mainfrom
perf/parallel-sql-macro-probes

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Why

Typed SQL-macro argument probes ran one model at a time on the session thread while the analysis pool sat idle: about half of analysis time in projects with many typed calls. Per-model analysis also repeated work: two SQLQuack analyses per model, a share key hashed from Debug text of every upstream column, and normalisation recounting characters from the start of the SQL per match.

Changes

  • Each model's probe plan, probes and findings run on the analysis pool; a probe's relation lookup reads a map built once per schema.
  • Lineage and name findings come from one statement_lineage_and_findings call.
  • The share key hashes its fields through Hash instead of Debug text.
  • Normalisation counts characters forward from the last offset (linear instead of quadratic).
  • Column nullability is read from a map built once per relation.
  • Tests: an E2E compile with typed calls in six waves, each model reporting its own finding; probe relation lookup; offsets of non-ASCII text across several edits.
  • Deferred: parallel Snowflake publish (it mutates session state per model), the second normalisation in share_key, Arc shapes, probe-name caching, pool size, probes skipping diagnostics.

Verification

Same orb, base main and this branch alternating, 3 runs each, load 1–3:

Project Output Wall median CPU median model_analysis_ms analysis CPU ms
1,000 models, 40 typed calls each, 5 waves identical 11.14 → 10.50 s 12.19 → 12.75 s 2,377 → 1,376 2,522 → 2,514
Dense 3,000 models identical 8.67 → 8.61 s 27.19 → 26.57 s 1,780 → 1,384 7,029 → 5,686

Compile JSON (minus timings) and every compiled file are byte-identical; shared/unshared analysis counts are unchanged (2,425 / 575, dense). cargo test -p sqlbuild-analysis, the SQL-macro E2E suite (301 passed), make check-ci passes; locally make test fails only the tests main fails here too (Docker, launcher version).

…op duplicate work

The probes that check each typed SQL-macro call ran one model at a time
on the session thread while the analysis pool sat idle; each is a parse
and a lineage analysis, so in projects with many typed calls they were
about half of all analysis time. Each model's probes now run on the
pool, and a probe's relation lookup reads a map built once per schema
instead of lower-casing, formatting and scanning every relation.

Duplicate work inside the per-model analysis is gone too:

- lineage and name findings come from one SQLQuack analysis instead of
  two;
- the share key feeds the hasher its fields instead of a Debug string of
  every upstream column;
- normalisation counts characters forward from the last edit instead of
  from the start of the SQL for every match, which was quadratic in
  long models;
- column nullability is read from a map instead of a scan per column.

Results are unchanged: findings, compile JSON and compiled files are
byte-identical before and after.
@kvlonge
kvlonge enabled auto-merge November 15, 2026 17:57
@kvlonge
kvlonge added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit d94e5b7 Nov 15, 2026
32 checks passed
@kvlonge
kvlonge deleted the perf/parallel-sql-macro-probes branch November 15, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant