Skip to content

docs: document DELETE and UPDATE for SQL users and table provider authors - #24567

Open
michaelsembwever wants to merge 1 commit into
apache:mainfrom
thelastpickle:mck/dml-delete-update-docs
Open

docs: document DELETE and UPDATE for SQL users and table provider authors#24567
michaelsembwever wants to merge 1 commit into
apache:mainfrom
thelastpickle:mck/dml-delete-update-docs

Conversation

@michaelsembwever

@michaelsembwever michaelsembwever commented Aug 21, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close relate to?

Rationale for this change

Since 52.0.0, DataFusion runs DELETE and UPDATE against a table whose provider implements TableProvider::delete_from() or TableProvider::update(), and the built-in in-memory table implements both. No page in the documentation says so.

A SQL user therefore cannot learn which tables accept the two statements, what a statement returns, or which forms fail. A provider author cannot learn what the planner passes to each hook, or what the hook must return.

Two current behaviours are surprising enough to warn about in the same pass:

  • A DELETE or an UPDATE whose WHERE clause holds an IN or an EXISTS subquery applies to all rows of the table. The optimizer rewrites the subquery into a LeftSemi Join, so extract_dml_filters() finds no predicate on the target table, and the provider reads the
    empty filter list as "no WHERE clause".

    > create table s1 as values (1), (2), (3);
    > create table s2 as values (2);
    > delete from s1 where column1 in (select column1 from s2);
    -- count 3; s1 is now empty
  • EXPLAIN DELETE and EXPLAIN UPDATE execute the statement on an in-memory table. MemTable changes the rows inside the hook, and the physical planner calls the hook while it builds the plan.

Both behaviours need code fixes, which this PR does not attempt. Until then a reader needs the warning.

What changes are included in this PR?

docs/source/user-guide/sql/dml.md:

  • A DELETE section and an UPDATE section: syntax, the count result, three-valued logic, and examples.
  • A "Table support for DELETE and UPDATE" section: which table kinds support the statements, and the exact error text for a table that does not.
  • A "Limitations" section: the two warnings above, the ignored LIMIT on DELETE, and UPDATE ... FROM.

docs/source/library-user-guide/custom-table-providers.md:

  • A "Row-Level DML: DELETE and UPDATE" section: what the planner passes to each hook (split AND conjunctions, stripped table qualifiers, target-table predicates only), the single-row count return contract, the two semantic rules a provider must follow, a compiling example,
    the clauses a hook never receives, and when the work happens.

No code changes.

Are these changes tested?

Yes.

  • cargo test --doc -p datafusion library_user_guide_custom_table_providers passes. The new example is a compiled doctest, not an ignore block.
  • ./ci/scripts/doc_prettier_check.sh passes.
  • Every behavioural statement in the new text was checked against main with temporary sqllogictest cases, rather than read from the code alone: the ignored LIMIT; the pre-statement values in SET a = b, b = a; the error text for an external table and for a view; the scalar
    subquery error; the IN and EXISTS all-rows result; and the EXPLAIN side effect. Those cases are not part of this PR, because the last two assert behaviour that should change.

Are there any user-facing changes?

Documentation only. No change to any API.

…hors

PR apache#19142 added `TableProvider::delete_from()` and `TableProvider::update()`,
and implemented both for `MemTable`, but added no documentation.

Add a `DELETE` section and an `UPDATE` section to the SQL user guide, with
the syntax, the result shape, which table kinds support the statements, and
the current limitations.

Add a "Row-Level DML" section to the custom table provider guide, covering
what the planner passes to each hook, the `count` result contract, the
semantic rules a provider must follow, and a compiling example.

Two behaviours found while verifying the documentation are recorded as
warnings, since users meet them today:

- An `IN` or an `EXISTS` subquery in the `WHERE` clause makes the statement
  apply to all rows, because the optimizer rewrites the subquery into a join
  and the predicate never reaches the provider.
- `EXPLAIN DELETE` and `EXPLAIN UPDATE` execute the statement on an
  in-memory table, because `MemTable` changes the rows inside the hook and
  the hook runs during physical planning.

Assisted-by: Claude Code:claude-opus-5
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 21, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.36%. Comparing base (a6e2d3f) to head (8dfa9c1).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24567      +/-   ##
==========================================
- Coverage   81.36%   81.36%   -0.01%     
==========================================
  Files        1117     1117              
  Lines      397872   397916      +44     
  Branches   397872   397916      +44     
==========================================
+ Hits       323725   323751      +26     
- Misses      55229    55243      +14     
- Partials    18918    18922       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support delete_from and update in TableProvider

2 participants