[Feat] Support Merge-on-Read mode for Deletes

[sungwy]

Feature Request / Improvement

Similar to Spark, we would like to implement Merge-on-Read mode of deleting, by creating delete files that encode position and equality delete markers that can be applied within a given partition.

https://iceberg.apache.org/spec/?h=delete+files#row-level-deletes

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[Fokko]

Or we can jump directly to deletion vectors 😏

[piyushdubey]

Reviving this thread. Is the support for Writing Position Deletes added to PyIceberg or not yet?

@Fokko, @sungwy

[Thomas-X]

Apologies for the necro-post, but is this supported yet? Writing to a normal sized iceberg table is painfully slow when forced to do CoW

[kevinjqliu]

i think we have all the necessary pieces to write v2 position delete, lets see if we can get this feature out.

We should think about what the default should be. If the table/file is small, it would be faster to just rewrite instead of writing delete files

[qzyu999]

Is this basically the same feature request for V2 positional deletes as tracked in #1808? Also, with V3 read support merged (#1516) and Deletion Vector write support underway (#2822), what is the long-term vision for maintaining both V2 and V3 architectures for MoR? Because V2 is highly integrated into stable enterprise pipelines while V3 is the future standard, it would make a lot of sense for PyIceberg to bridge and maintain both capabilities.

[moomindani]

Regarding the v3 side of this issue: the v3 spec requires position deletes to be written as deletion vectors in Puffin files, so the merge-on-read DELETE path needs a DV writer rather than position delete files. As @Fokko mentioned earlier in this thread, jumping directly to deletion vectors is an option — for v3 tables it is the required one, while the v2 position-delete-file discussion above stays independent of this.

I have been working on building blocks for the v3 part (continuing #2822, coordinated with its original authors), and here is how I am currently thinking about it:

1. Add PuffinWriter for writing deletion vectors #3474 (under review) adds a low-level PuffinWriter that serializes deletion-vector-v1 blobs, and Add Spark interop test for reading Puffin deletion vectors #3476 adds an integration test verifying PyIceberg can read Spark-written DVs.
2. Next, extend PuffinWriter to write one DV blob per referenced data file (mirroring Java's BaseDVFileWriter) and expose blob offsets/lengths, which are needed for DataFile's content_offset / content_size_in_bytes.
3. Then wire this into the merge-on-read branch of Transaction.delete() for v3 tables: compute matched row positions per data file, union with an existing DV when present (at most one DV per data file), write one Puffin file per commit, and add the DV entries through a delete snapshot. Spark interop tests (Spark reading PyIceberg-written DVs) would come with this step.

On the default question @kevinjqliu raised above: in this plan write.delete.mode=merge-on-read would only take effect on v3 tables, and copy-on-write stays the default for everything, so behavior does not change unless a user opts in.

This is the approach I currently have in mind, but I am open to alternatives, a different split of the work, or aligning with anything already planned by the maintainers. Any feedback before I move forward with the follow-up PRs is very welcome.