Skip to content

feat(motherduck): read Parquet files in S3 on MotherDuck's servers - #344

Merged
kvlonge merged 4 commits into
mainfrom
motherduck-s3
Nov 18, 2026
Merged

kvlonge merged 4 commits into
mainfrom
motherduck-s3

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Parquet files of a files source in S3 now load into MotherDuck straight from the bucket: MotherDuck's servers read them with read_parquet (MD_RUN = REMOTE), and no row passes through SQLLoad.

  • Credentials. For each bucket, MotherDuck gets a temporary secret that lasts only for the load's session, made from the keys SQLLoad finds for the s3 connection: the environment, else the shared credentials file, else the AWS credential chain resolved on this machine. Secrets are cached by bucket and set again when their keys change. A chain secret is set again after four minutes, because role keys may expire. Secrets are set before a table's transaction begins, never inside it. Nothing is stored in the MotherDuck account. A narrower secret of the user's own in MotherDuck still wins.
  • Check at plan. plan reads the first file to load on MotherDuck's servers, with the same read_parquet options as the load. It fails the table with the fix when MotherDuck can't read it, for example a bucket whose policy allows only the user's network or VPC endpoint, or keys without kms:Decrypt.
  • Loading. The route loads the same values as SQLLoad's reader, lineage columns included. A batch fails unless MotherDuck reports inserting as many rows as the files' footers hold.
  • What still goes through SQLLoad. These still go through SQLLoad, as on DuckDB: CSV and JSON files, masked columns, a custom S3 endpoint, files in a local folder or over SFTP, and a Snowflake source's unloaded files.

The docs cover this in the MotherDuck target page ("Parquet files in S3"), the DuckDB and targets overview pages, the supported-systems table and the landing page.

Breaking change

Parquet files in S3 that MotherDuck's servers can't read now fail plan, naming the bucket and the fix. They used to load through SQLLoad. There is no setting to keep that route.

Verification

  • Unit tests cover the generated read_parquet and probe SQL, the secrets per bucket (hash, chain refresh and a user's narrower secret), and the route reasons.
  • Live, against a MotherDuck account reading a real S3 bucket: append, replace, and delete_insert on a partition cursor and by replace_key, with tables loading at the same time. Each load was compared row for row (EXCEPT ALL both ways, lineage columns included) with the same files read by SQLLoad, and then run a second time. A bucket only this machine may read failed plan with the fix.
  • make verify and make test-python (609 passed) pass on the final head.

…, MD_RUN = REMOTE, writer accepts s3 Parquet
… file there, secrets set once per database before a table's transaction, Snowflake unloads stay on SQLLoad's reader, docs
… hash, chain secrets set again after ten minutes, no secret inside a transaction, network fix text, probe matches the load
@kvlonge
kvlonge merged commit 4cab9f2 into main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant