Skip to content

feat(snowflake): Databricks and ClickHouse read a Snowflake S3 unload themselves - #346

Merged
kvlonge merged 7 commits into
mainfrom
snowflake-unload-reads
Nov 18, 2026
Merged

kvlonge merged 7 commits into
mainfrom
snowflake-unload-reads

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

A Snowflake source with a stage url in S3 now loads into Databricks and ClickHouse without any row passing through SQLLoad, as it already did into DuckDB.

  • Databricks COPYs the files Snowflake unloads through the Unity Catalog external location that covers the url, into a staging or build table (an append too). A batch publishes only when the COPYs load as many rows as Snowflake reported unloading.
  • ClickHouse reads each unloaded file with s3() into the load's staging table, signed with the same S3 credentials as SQLLoad, and publishes only when it loads as many rows as were unloaded.
  • plan shows the route (unload Parquet to s3://…/ -> Databricks COPY INTO (nothing through the loader)). A table with a TIME column or a mask stays on SQLLoad's reader, and plan says why.
  • Each unload is its own batch, so merge row numbers stay unique across cursor windows (this also fixes the same-connection Snowflake route). An unloaded table's read takes at most 32,767 cursor windows, so those row numbers fit in 64 bits; more fails plan, asking for a larger batch_size.
  • The planned unload store ends in /, so an external location over exactly that folder covers it.

BREAKING CHANGE: a Snowflake S3 unload into Databricks that no external location covers now fails plan with the fix, instead of loading through SQLLoad. A Snowflake table read through an unload is limited to 32,767 cursor windows.

Testing

  • Unit tests for the ClickHouse and Databricks unload statements, the routing and coverage checks, the window limit and the store URL.
  • make verify and make test-python pass on this head.
  • Live, Snowflake to a ClickHouse container reading real S3, and Snowflake to Databricks through an external location: replace and append match the loader route row for row (EXCEPT ALL 0/0 both ways, 3,000 rows including pre-1970 timestamps). A rerun works, a url outside every location fails plan with the fix, and the unload folder is empty afterwards. Merge loads across windows: 99 stale keys before the batch fix, 0 after.

@kvlonge
kvlonge merged commit fac8e30 into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant