Skip to content

feat(snowflake): load a Snowflake source into Snowflake without rows passing through SQLLoad - #337

Merged
kvlonge merged 9 commits into
mainfrom
snowflake-into-snowflake
Nov 18, 2026
Merged

kvlonge merged 9 commits into
mainfrom
snowflake-into-snowflake

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

A Snowflake source loading into a Snowflake target no longer sends its rows through SQLLoad in two cases.

  • Same connection. The target writes through the source's own connection, with no other database, and the source sets no warehouse of its own. Snowflake runs one INSERT ... SELECT on that connection, and nothing is unloaded. plan shows route Snowflake INSERT ... SELECT on the source's connection (nothing through the loader).
  • Unload inside the target's external stage. The source's stage url is inside the target's external stage url, on another connection or account. Snowflake unloads Parquet there, and the target COPYs the files through its own stage. The batch commits only when the rows loaded add up to what the unload reported. A COPY whose FILES list is longer than one statement allows is split across requests. Each request commits into the load's own staging or build table, and the table publishes only when the sum matches. An append that writes straight into its table can't be split, so when the unload is too large it fails before loading anything, and the error suggests a smaller batch_size.

In every other case nothing changes: the user stage, a named stage, an S3 folder the target's stage doesn't cover, or a table with a mask. SQLLoad reads the unloaded files and stages them as before, and plan says why.

Times and timestamps are unloaded as whole microseconds and turned back exactly, as with DuckDB. A merge's row number comes from the file's position in the unload and the row's position in the file. A file Snowflake reads that isn't in the unload fails the COPY and names the file.

The docs cover this in the Snowflake source page ("Snowflake loads the rows itself"), the Snowflake and target overview pages, the supported-systems table and the landing page.

Verification

  • Unit tests cover the generated SQL, the stage-cover rule (folder edges, other buckets, user and named stages, masks) and how the requests are split.
  • Live, against a real Snowflake account and an S3 bucket behind a storage integration: both routes ran a first load and an incremental merge, and MINUS both ways between source and target was empty. The test account resources were removed afterwards.
  • make verify and make test-python pass on the final head.

…ernal stage

Incomplete: the cross-account COPY of unloaded files and its SQL test.
Not yet run against Snowflake; the same-connection route is not started.
…onnection

The destination loads a Snowflake source's rows inside Snowflake: an
INSERT ... SELECT of the unload's query when the target writes through
the source's own connection and database, otherwise COPYs of the
unloaded files from the target's external stage, committed only when
their rows match the unload's. Not yet run against Snowflake.
…ording for selected rows

Live check on Snowflake: same-connection INSERT ... SELECT and the
through-SQLLoad route load identical values (decimals, zoned and plain
timestamps to the microsecond, times, VARIANT), first run and an
incremental merge after the cursor.
…entry module, branch-free tests, loader files under the line limit
@kvlonge
kvlonge merged commit 7dc9852 into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant