Repository navigation
feat(gcs): Google Cloud Storage files sources and targets, read directly by DuckDB and MotherDuck - #350
Merged
Conversation
…count key or application default credentials, gs:// paths
…entials, DuckDB and MotherDuck read gs:// Parquet with the connection's HMAC key
…eading GCS with an HMAC key
Parquet append, pattern with order, replace_key with on_overlapping_files, files target strategies, and after_load move and delete all run against a GCS server emulator. The GCS client builder moves beside the credential chain it reads.
…ad itself A plan of one table shows the reason on its why line, such as a gcs connection without an HMAC key. The DuckDB and MotherDuck fix for an unreadable GCS file names a wrong or inactive HMAC key as well as the bucket role. Docs cover GCS in the files guide and the files target.
…on sources - A GOOGLE_APPLICATION_CREDENTIALS that names no file fails the GCS client instead of falling through to other credentials. - A Python source may no longer share a files target's gcs connection, which holds the bucket's keys; s3 and filesystem connections still may. - Databricks reads files itself only under S3 external locations. - The route reason names what keeps a destination from reading files only when it could read them, and the --reload note for files after_load moved stays on the route line.
Deploying sqlload with
|
| Latest commit: |
139364b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://b8e87790.sqlload.pages.dev |
| Branch Preview URL: | https://gcs-support.sqlload.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Google Cloud Storage joins S3 as a place for files: a
gcsconnection,gs://files sources and files targets, and DuckDB and MotherDuck reading Parquet in GCS themselves.gcsconnection.bucket, optionalservice_account_key, optionalhmac_key_idandhmac_secret(both or neither), andendpointfor an emulator. Without a key, SQLLoad uses Google's application default credentials: the fileGOOGLE_APPLICATION_CREDENTIALSnames, gcloud's file, else the machine's service account.sqlload debugsays which one each connection uses, never the credentials.pattern,order,replace_key,on_overlapping_files,after_load(move_toanddelete), and the files target'sappend,replaceanddelete_insert. Amove_tocopies server-side and only when no file is at the destination, so a move never overwrites.gs://Parquet themselves, under the same rules as S3, when the connection has an HMAC key: they read GCS only with one, not with a service-account key. The key goes into a temporarygcssecret scoped to the bucket, passed as parameters. Without an HMAC key, or with a customendpoint, SQLLoad reads the files instead.planreads the first file to load with DuckDB or MotherDuck and fails the table with the fix when it can't.sqlload plan -v <table>says why SQLLoad reads Parquet a destination could have read itself, such as agcsconnection without an HMAC key or a masked column. Before, only staged warehouse routes showed the reason. A destination that can't read files at all gets the plain "cannot read the files where they are" reason.GOOGLE_APPLICATION_CREDENTIALSthat names no file fails the load instead of falling back to other credentials. A Python source can't share a files target'sgcsconnection, which holds the bucket's keys;s3andfilesystemconnections still can. Databricks reads files itself only under S3 external locations, where SQLLoad checks thatCOPY INTOkeeps every value.Testing
patternwithorder = "path",replace_keywithon_overlapping_files = "last_wins", the files target's three strategies, andafter_loadmove and delete. Each fails withoutgcsin the files connection kinds.whyline, a Python source on agcstarget connection, and Databricks' S3-only locations.gs://directly and SQLLoad's reader loaded identical rows (EXCEPT ALLboth ways: 0 and 0).gs://on its servers forappendanddelete_insert, identical to SQLLoad's reader; a rerun added no duplicates.planon DuckDB and on MotherDuck, with the fix.delete_insertwrote one file per day, and a rerun replaced only the newest day.after_load:deleteremoved the file,move_tomoved one, and a move onto an existing file was refused (412) and left both files as they were.pattern,orderandlast_winsgave the expected rows;GOOGLE_APPLICATION_CREDENTIALSworked as the only credential, and one naming a missing file failedplanwith the path.make verifyandmake test-pythonpass on the final head.