Repository navigation
feat(motherduck): read Parquet files in S3 on MotherDuck's servers - #344
Merged
Merged
Conversation
…, MD_RUN = REMOTE, writer accepts s3 Parquet
… file there, secrets set once per database before a table's transaction, Snowflake unloads stay on SQLLoad's reader, docs
… hash, chain secrets set again after ten minutes, no secret inside a transaction, network fix text, probe matches the load
…tes role keys are promised
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parquet files of a files source in S3 now load into MotherDuck straight from the bucket: MotherDuck's servers read them with
read_parquet(MD_RUN = REMOTE), and no row passes through SQLLoad.s3connection: the environment, else the shared credentials file, else the AWS credential chain resolved on this machine. Secrets are cached by bucket and set again when their keys change. A chain secret is set again after four minutes, because role keys may expire. Secrets are set before a table's transaction begins, never inside it. Nothing is stored in the MotherDuck account. A narrower secret of the user's own in MotherDuck still wins.planreads the first file to load on MotherDuck's servers, with the sameread_parquetoptions as the load. It fails the table with the fix when MotherDuck can't read it, for example a bucket whose policy allows only the user's network or VPC endpoint, or keys withoutkms:Decrypt.The docs cover this in the MotherDuck target page ("Parquet files in S3"), the DuckDB and targets overview pages, the supported-systems table and the landing page.
Breaking change
Parquet files in S3 that MotherDuck's servers can't read now fail
plan, naming the bucket and the fix. They used to load through SQLLoad. There is no setting to keep that route.Verification
read_parquetand probe SQL, the secrets per bucket (hash, chain refresh and a user's narrower secret), and the route reasons.append,replace, anddelete_inserton a partition cursor and byreplace_key, with tables loading at the same time. Each load was compared row for row (EXCEPT ALLboth ways, lineage columns included) with the same files read by SQLLoad, and then run a second time. A bucket only this machine may read failedplanwith the fix.make verifyandmake test-python(609 passed) pass on the final head.