Skip to content

feat(gcs): Google Cloud Storage files sources and targets, read directly by DuckDB and MotherDuck - #350

Merged
kvlonge merged 9 commits into
mainfrom
gcs-support
Nov 19, 2026
Merged

kvlonge merged 9 commits into
mainfrom
gcs-support

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Google Cloud Storage joins S3 as a place for files: a gcs connection, gs:// files sources and files targets, and DuckDB and MotherDuck reading Parquet in GCS themselves.

  • gcs connection. bucket, optional service_account_key, optional hmac_key_id and hmac_secret (both or neither), and endpoint for an emulator. Without a key, SQLLoad uses Google's application default credentials: the file GOOGLE_APPLICATION_CREDENTIALS names, gcloud's file, else the machine's service account. sqlload debug says which one each connection uses, never the credentials.
  • Files sources and targets in GCS, with what S3 has: pattern, order, replace_key, on_overlapping_files, after_load (move_to and delete), and the files target's append, replace and delete_insert. A move_to copies server-side and only when no file is at the destination, so a move never overwrites.
  • DuckDB and MotherDuck read gs:// Parquet themselves, under the same rules as S3, when the connection has an HMAC key: they read GCS only with one, not with a service-account key. The key goes into a temporary gcs secret scoped to the bucket, passed as parameters. Without an HMAC key, or with a custom endpoint, SQLLoad reads the files instead. plan reads the first file to load with DuckDB or MotherDuck and fails the table with the fix when it can't.
  • sqlload plan -v <table> says why SQLLoad reads Parquet a destination could have read itself, such as a gcs connection without an HMAC key or a masked column. Before, only staged warehouse routes showed the reason. A destination that can't read files at all gets the plain "cannot read the files where they are" reason.
  • Credentials stay where they belong. A GOOGLE_APPLICATION_CREDENTIALS that names no file fails the load instead of falling back to other credentials. A Python source can't share a files target's gcs connection, which holds the bucket's keys; s3 and filesystem connections still can. Databricks reads files itself only under S3 external locations, where SQLLoad checks that COPY INTO keeps every value.

Testing

  • Loader tests against a GCS emulator: Parquet append, pattern with order = "path", replace_key with on_overlapping_files = "last_wins", the files target's three strategies, and after_load move and delete. Each fails without gcs in the files connection kinds.
  • Unit tests: the credential chain order and a missing key file, the DuckDB secret for an HMAC key, routing without one, the route reason and the why line, a Python source on a gcs target connection, and Databricks' S3-only locations.
  • Live, against a real bucket:
    • DuckDB reading gs:// directly and SQLLoad's reader loaded identical rows (EXCEPT ALL both ways: 0 and 0).
    • MotherDuck read gs:// on its servers for append and delete_insert, identical to SQLLoad's reader; a rerun added no duplicates.
    • A wrong HMAC secret fails plan on DuckDB and on MotherDuck, with the fix.
    • A files target in GCS: delete_insert wrote one file per day, and a rerun replaced only the newest day.
    • after_load: delete removed the file, move_to moved one, and a move onto an existing file was refused (412) and left both files as they were.
    • pattern, order and last_wins gave the expected rows; GOOGLE_APPLICATION_CREDENTIALS worked as the only credential, and one naming a missing file failed plan with the path.
  • gcloud's application default credentials and a machine's service account weren't tested live, so the docs mark them (Beta).
  • make verify and make test-python pass on the final head.

…count key or application default credentials, gs:// paths
…entials, DuckDB and MotherDuck read gs:// Parquet with the connection's HMAC key
Parquet append, pattern with order, replace_key with on_overlapping_files,
files target strategies, and after_load move and delete all run against a
GCS server emulator. The GCS client builder moves beside the credential
chain it reads.
…ad itself

A plan of one table shows the reason on its why line, such as a gcs
connection without an HMAC key. The DuckDB and MotherDuck fix for an
unreadable GCS file names a wrong or inactive HMAC key as well as the
bucket role. Docs cover GCS in the files guide and the files target.
…on sources

- A GOOGLE_APPLICATION_CREDENTIALS that names no file fails the GCS
  client instead of falling through to other credentials.
- A Python source may no longer share a files target's gcs connection,
  which holds the bucket's keys; s3 and filesystem connections still may.
- Databricks reads files itself only under S3 external locations.
- The route reason names what keeps a destination from reading files
  only when it could read them, and the --reload note for files
  after_load moved stays on the route line.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Deploying sqlload with  Cloudflare Pages  Cloudflare Pages

Latest commit: 139364b
Status: ✅  Deploy successful!
Preview URL: https://b8e87790.sqlload.pages.dev
Branch Preview URL: https://gcs-support.sqlload.pages.dev

View logs

@kvlonge
kvlonge merged commit 757ec17 into main Nov 19, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant