Skip to content

fix: plan from the compile a worker verified, and load seeds from a shared queue - #1572

Merged
kvlonge merged 3 commits into
mainfrom
fix/worker-compile-once-seed-queue
Nov 18, 2026
Merged

kvlonge merged 3 commits into
mainfrom
fix/worker-compile-once-seed-queue

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Why

A location worker compiled the project to verify it, threw that compile away, and the build compiled it again before planning. That added about 7 s to every run's start-up on a large project, and more under load. sqb seed also handed seeds to connections round-robin up front, so a slow seed held up every seed queued behind it on its connection, and each seed load issued statements it did not need.

Changes

  • A worker hands the compile it verified to the build, which plans from it. The build still compiles itself when its compile could differ: when the run has variables, SQL validation is off, the cache is off, a dbt manifest resolves references, or the connection comes from a dbt profile.
  • sqb seed connections take the next seed from one shared queue. Builds already did this.
  • A Snowflake seed load drops a replaced table with one DROP TABLE IF EXISTS instead of also dropping a view, with the same error as before when the destination is a view.
  • Seeds share the run's fingerprint-table setup, as models do: the first writer sets the table up, and later writers insert directly. A 93-seed run issues 442 statements instead of 627.

Verification

  • Worker integration tests: the verified compile is reused for a plain job, and the build compiles again for run variables, SQL validation off and the cache off. Not covered by a test: the dbt manifest and dbt-profile exclusions.
  • Seeding tests: a slow seed no longer delays the others (round-robin times out), and the fingerprint table is set up once (4 times before).
  • Snowflake client test: one table drop per replaced seed.
  • No DuckShop edge case: these change how a worker and sqb seed schedule work, not a project shape, data value or model lifecycle.
  • make check-ci passed on the final head.

…y itself

sqb seed assigned seeds to its connections round-robin up front, so the
seeds behind a slow one on the same connection waited for it while other
connections sat idle. Each connection now takes the next seed from one
queue. Builds already schedule seeds from a shared ready queue.
…piling again

A worker compiled the project to verify it matches the published
snapshot, then threw that compile away, and the build compiled the
project a second time before planning: about 7 s of every run's
start-up on a 2,280-model project, and twice that under load.

The worker now hands its verified compile to the build, which plans from
it as a location's plan preview does. A build whose compile could differ
still compiles: one with run variables, SQL validation off or a dbt
manifest to resolve references from.
…ble up once per run

Each Snowflake seed load ran DROP TABLE IF EXISTS and then DROP VIEW IF
EXISTS. The second can never do anything: when the first succeeds no view
of that name exists, and when the destination is a view the first already
fails. Seed loads now issue only the table drop, with the same error for a
view destination as before.

Each seed also set up the fingerprint table before writing its row. Seeds
in a build or a seed run now share the run's fingerprint writes, as models
do: the first writer sets the table up, concurrent writers wait for it, and
later writers insert directly and set the table up only if the insert
fails. Every seed still records its row. A 93-seed run issues 442
statements instead of 627.
@kvlonge
kvlonge marked this pull request as ready for review November 18, 2026 11:43
@kvlonge
kvlonge added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit eefc833 Nov 18, 2026
60 of 62 checks passed
@kvlonge
kvlonge deleted the fix/worker-compile-once-seed-queue branch November 18, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant