Repository navigation
fix: plan from the compile a worker verified, and load seeds from a shared queue - #1572
Merged
Merged
Conversation
…y itself sqb seed assigned seeds to its connections round-robin up front, so the seeds behind a slow one on the same connection waited for it while other connections sat idle. Each connection now takes the next seed from one queue. Builds already schedule seeds from a shared ready queue.
…piling again A worker compiled the project to verify it matches the published snapshot, then threw that compile away, and the build compiled the project a second time before planning: about 7 s of every run's start-up on a 2,280-model project, and twice that under load. The worker now hands its verified compile to the build, which plans from it as a location's plan preview does. A build whose compile could differ still compiles: one with run variables, SQL validation off or a dbt manifest to resolve references from.
…ble up once per run Each Snowflake seed load ran DROP TABLE IF EXISTS and then DROP VIEW IF EXISTS. The second can never do anything: when the first succeeds no view of that name exists, and when the destination is a view the first already fails. Seed loads now issue only the table drop, with the same error for a view destination as before. Each seed also set up the fingerprint table before writing its row. Seeds in a build or a seed run now share the run's fingerprint writes, as models do: the first writer sets the table up, concurrent writers wait for it, and later writers insert directly and set the table up only if the insert fails. Every seed still records its row. A 93-seed run issues 442 statements instead of 627.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A location worker compiled the project to verify it, threw that compile away, and the build compiled it again before planning. That added about 7 s to every run's start-up on a large project, and more under load.
sqb seedalso handed seeds to connections round-robin up front, so a slow seed held up every seed queued behind it on its connection, and each seed load issued statements it did not need.Changes
sqb seedconnections take the next seed from one shared queue. Builds already did this.DROP TABLE IF EXISTSinstead of also dropping a view, with the same error as before when the destination is a view.Verification
sqb seedschedule work, not a project shape, data value or model lifecycle.make check-cipassed on the final head.