Repository navigation
feat(snowflake): load a Snowflake source into Snowflake without rows passing through SQLLoad - #337
Merged
Merged
Conversation
…ernal stage Incomplete: the cross-account COPY of unloaded files and its SQL test. Not yet run against Snowflake; the same-connection route is not started.
…onnection The destination loads a Snowflake source's rows inside Snowflake: an INSERT ... SELECT of the unload's query when the target writes through the source's own connection and database, otherwise COPYs of the unloaded files from the target's external stage, committed only when their rows match the unload's. Not yet run against Snowflake.
…ording for selected rows Live check on Snowflake: same-connection INSERT ... SELECT and the through-SQLLoad route load identical values (decimals, zoned and plain timestamps to the microsecond, times, VARIANT), first run and an incremental merge after the cursor.
…t's external stage covers it
…limit, unexpected file, doc scope
…al for a direct append
…entry module, branch-free tests, loader files under the line limit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Snowflake source loading into a Snowflake target no longer sends its rows through SQLLoad in two cases.
database, and the source sets nowarehouseof its own. Snowflake runs oneINSERT ... SELECTon that connection, and nothing is unloaded.planshowsroute Snowflake INSERT ... SELECT on the source's connection (nothing through the loader).stageurlis inside the target's external stageurl, on another connection or account. Snowflake unloads Parquet there, and the targetCOPYs the files through its own stage. The batch commits only when the rows loaded add up to what the unload reported. ACOPYwhoseFILESlist is longer than one statement allows is split across requests. Each request commits into the load's own staging or build table, and the table publishes only when the sum matches. Anappendthat writes straight into its table can't be split, so when the unload is too large it fails before loading anything, and the error suggests a smallerbatch_size.In every other case nothing changes: the user stage, a named stage, an S3 folder the target's stage doesn't cover, or a table with a
mask. SQLLoad reads the unloaded files and stages them as before, andplansays why.Times and timestamps are unloaded as whole microseconds and turned back exactly, as with DuckDB. A merge's row number comes from the file's position in the unload and the row's position in the file. A file Snowflake reads that isn't in the unload fails the
COPYand names the file.The docs cover this in the Snowflake source page ("Snowflake loads the rows itself"), the Snowflake and target overview pages, the supported-systems table and the landing page.
Verification
MINUSboth ways between source and target was empty. The test account resources were removed afterwards.make verifyandmake test-pythonpass on the final head.