Repository navigation
feat: read DuckDB files and MotherDuck databases as sources - #330
Merged
Merged
Conversation
…s, odbc reader moved to its source
A duckdb connection's file is read read-only. A motherduck connection is read through the MotherDuck bridge, which streams query results as Arrow.
…otherDuck source bridges in the memory budget Reads of one file share one read-only instance, so ending one read no longer drops the process's lock on the file, and spills go to a temporary folder instead of beside the file. A MotherDuck source's bridge process is counted by the memory watchdog while it is open. The bridge never writes a batch file SQLLoad has already removed.
Reads of one file sharing an instance now get the limit separate instances had between them, and an instance whose first connection fails is never left open.
The catalog's primary-key query used to_json, which DuckDB's bundled build doesn't include, so listing a file's tables downloaded the json extension first. That failed offline, and on Windows two reads installing it at once collided. The query now returns one key column a row, in key order, using only DuckDB's own functions. The lock test also counts Windows' sharing violation as the lock.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two source kinds:
duckdb, which reads a DuckDB database file, andmotherduck, which reads a MotherDuck database.DuckDB source
includeandexclude, plus query tables (sql), with the catalog's primary keys and comments. Cursors,whereandbatch_sizework as for the other SQL sources.memory_limitof half a table's share.:memory:is refused, and so is anmd:path on aduckdbconnection, which is amotherducksource.MotherDuck source (Beta)
streamandnextoperations, which hand each batch to SQLLoad as an Arrow IPC file. They useto_arrow_reader, orfetch_record_batchon duckdb 1.4.motherduckextra now installspyarrowtoo.tests/test_motherduck_live.py, including a new source test), as well as with the offline stand-in intests/test_motherduck.py.Other changes
Verification
make verifyon this head: fmt, the clippy variants, fensu, 383 unit tests, the CLI tests, and the container suites (17 exporter, 135 loader, including the new DuckDB source test). All pass.make test-python: 609 passed, 3 skipped.tests/test_motherduck_live.pyagainst a real MotherDuck account: 3 passed, including the new source test.memory_limitfor each open read;write_nextbefore and after the change.