docs: document DELETE and UPDATE for SQL users and table provider authors - #24567
Open
michaelsembwever wants to merge 1 commit into
Open
docs: document DELETE and UPDATE for SQL users and table provider authors#24567michaelsembwever wants to merge 1 commit into
michaelsembwever wants to merge 1 commit into
Conversation
…hors PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`, and implemented both for `MemTable`, but added no documentation. Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with the syntax, the result shape, which table kinds support the statements, and the current limitations. Add a "Row-Level DML" section to the custom table provider guide, covering what the planner passes to each hook, the `count` result contract, the semantic rules a provider must follow, and a compiling example. Two behaviours found while verifying the documentation are recorded as warnings, since users meet them today: - An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement apply to all rows, because the optimizer rewrites the subquery into a join and the predicate never reaches the provider. - `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an in-memory table, because `MemTable` changes the rows inside the hook and the hook runs during physical planning. Assisted-by: Claude Code:claude-opus-5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24567 +/- ##
==========================================
- Coverage 81.36% 81.36% -0.01%
==========================================
Files 1117 1117
Lines 397872 397916 +44
Branches 397872 397916 +44
==========================================
+ Hits 323725 323751 +26
- Misses 55229 55243 +14
- Partials 18918 18922 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR
closerelate to?TableProviderrow-level DML hooks and closed Supportdelete_fromandupdateinTableProvider#16959. That PR shipped no documentation.UPDATE ...FROMbug #19950.Rationale for this change
Since 52.0.0, DataFusion runs
DELETEandUPDATEagainst a table whose provider implementsTableProvider::delete_from()orTableProvider::update(), and the built-in in-memory table implements both. No page in the documentation says so.A SQL user therefore cannot learn which tables accept the two statements, what a statement returns, or which forms fail. A provider author cannot learn what the planner passes to each hook, or what the hook must return.
Two current behaviours are surprising enough to warn about in the same pass:
A
DELETEor anUPDATEwhoseWHEREclause holds anINor anEXISTSsubquery applies to all rows of the table. The optimizer rewrites the subquery into aLeftSemi Join, soextract_dml_filters()finds no predicate on the target table, and the provider reads theempty filter list as "no
WHEREclause".EXPLAIN DELETEandEXPLAIN UPDATEexecute the statement on an in-memory table.MemTablechanges the rows inside the hook, and the physical planner calls the hook while it builds the plan.Both behaviours need code fixes, which this PR does not attempt. Until then a reader needs the warning.
What changes are included in this PR?
docs/source/user-guide/sql/dml.md:DELETEsection and anUPDATEsection: syntax, thecountresult, three-valued logic, and examples.LIMITonDELETE, andUPDATE ... FROM.docs/source/library-user-guide/custom-table-providers.md:ANDconjunctions, stripped table qualifiers, target-table predicates only), the single-rowcountreturn contract, the two semantic rules a provider must follow, a compiling example,the clauses a hook never receives, and when the work happens.
No code changes.
Are these changes tested?
Yes.
cargo test --doc -p datafusion library_user_guide_custom_table_providerspasses. The new example is a compiled doctest, not anignoreblock../ci/scripts/doc_prettier_check.shpasses.mainwith temporary sqllogictest cases, rather than read from the code alone: the ignoredLIMIT; the pre-statement values inSET a = b, b = a; the error text for an external table and for a view; the scalarsubquery error; the
INandEXISTSall-rows result; and theEXPLAINside effect. Those cases are not part of this PR, because the last two assert behaviour that should change.Are there any user-facing changes?
Documentation only. No change to any API.