Skip to content

feat: read DuckDB files and MotherDuck databases as sources - #330

Merged
kvlonge merged 9 commits into
mainfrom
duckdb-source
Nov 18, 2026
Merged

kvlonge merged 9 commits into
mainfrom
duckdb-source

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Adds two source kinds: duckdb, which reads a DuckDB database file, and motherduck, which reads a MotherDuck database.

DuckDB source

  • Tables and views found by include and exclude, plus query tables (sql), with the catalog's primary keys and comments. Cursors, where and batch_size work as for the other SQL sources.
  • The file is opened read-only, so SQLLoad never changes it and holds no write lock on it. Reads open at the same time share one read-only instance, so the file stays locked against writers until the last read ends. Spills go to a temporary folder of SQLLoad's, not beside the file.
  • Under a memory budget, each read gets a memory_limit of half a table's share.
  • :memory: is refused, and so is an md: path on a duckdb connection, which is a motherduck source.

MotherDuck source (Beta)

  • Reads through the same separate Python process a MotherDuck target uses, because MotherDuck doesn't accept the DuckDB that SQLLoad bundles yet. That process gains stream and next operations, which hand each batch to SQLLoad as an Arrow IPC file. They use to_arrow_reader, or fetch_record_batch on duckdb 1.4.
  • The motherduck extra now installs pyarrow too.
  • The connection must name one database. The token is handled as for a MotherDuck target and never appears in messages.
  • Under a memory budget, the source's process is counted by the memory watchdog while it is open.
  • Tested live against a real MotherDuck account (tests/test_motherduck_live.py, including a new source test), as well as with the offline stand-in in tests/test_motherduck.py.

Other changes

  • Row counts for every source kind moved into one place, used for both the plan and a failed load's count.
  • The ODBC reader moved under the ODBC source, and the Snowflake source's debug check moved under the Snowflake source. Messages and SQL are unchanged.
  • Docs: new DuckDB and MotherDuck source pages, the sources index, supported systems, configuration and the MotherDuck target's install note.

Verification

  • make verify on this head: fmt, the clippy variants, fensu, 383 unit tests, the CLI tests, and the container suites (17 exporter, 135 loader, including the new DuckDB source test). All pass.
  • make test-python: 609 passed, 3 skipped.
  • tests/test_motherduck_live.py against a real MotherDuck account: 3 passed, including the new source test.
  • Each fix from review has a test that fails without it:
    • a second process can't write a DuckDB file while any read of it is open;
    • a shared instance gets one read's memory_limit for each open read;
    • the watchdog stops counting a bridge once it ends.
  • A removed batch file is no longer recreated by the bridge. This was checked by running write_next before and after the change.

A duckdb connection's file is read read-only. A motherduck connection is
read through the MotherDuck bridge, which streams query results as Arrow.
…otherDuck source bridges in the memory budget

Reads of one file share one read-only instance, so ending one read no
longer drops the process's lock on the file, and spills go to a temporary
folder instead of beside the file. A MotherDuck source's bridge process is
counted by the memory watchdog while it is open. The bridge never writes a
batch file SQLLoad has already removed.
Reads of one file sharing an instance now get the limit separate instances
had between them, and an instance whose first connection fails is never left
open.
The catalog's primary-key query used to_json, which DuckDB's bundled
build doesn't include, so listing a file's tables downloaded the json
extension first. That failed offline, and on Windows two reads
installing it at once collided. The query now returns one key column a
row, in key order, using only DuckDB's own functions.

The lock test also counts Windows' sharing violation as the lock.
@kvlonge
kvlonge merged commit 0be3344 into main Nov 18, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant