Core: V4 write direction wrappers by stevenzwu · Pull Request #16936 · apache/iceberg

stevenzwu · 2026-06-23T06:05:07Z

What

Adds v4 write-direction TrackedFile adapters: reusable forwarding wrappers that present a legacy v2/v3 ContentFile (DataFile/DeleteFile) as a v4 manifest-entry row on the write path, without materializing a fresh struct per row. A single adapter is allocated per writer and re-pointed at each file; content stats are served by a map-backed view (MapBackedContentStats) over the file's stat maps, with copy() producing stable snapshots.

Benchmark summary

JMH microbenchmarks compared two strategies for presenting a legacy file as a v4 row: convert (materialize a fresh struct per row) vs wrap (a reusable wrapper allocated once per writer and re-pointed per row, with zero per-row allocation). Throughput in ops/ms (higher is better); allocation via the GC profiler in B/op (lower is better).

Per-column content stats: wrap runs ~1.6–1.9x faster than convert across all column counts and allocates far less (39,280 vs 98,984 B/op at 200 columns). Convert pays per-column allocation and eager bound decode that the map-backed view avoids.
Fixed envelope (null stats): convert and wrap are within measurement error (9130 vs 8680 ops/ms); wrap still allocates less (256 vs 688 B/op). The envelope's small, fixed field count means the wrapper's positional dispatch neither meaningfully helps nor hurts.
Combined (envelope + stats): wrap-both is fastest or statistically tied at every column count with the lowest per-row allocation. The column-stats cost dominates and scales with table width, so wrapping stats is what matters; the fixed envelope edge is swamped once real column stats are present.

Net: the reusable wrap model is adopted because it wins or ties everywhere and minimizes per-row allocation — the cost that grows with table width.

See the benchmark doc for full methodology and per-column numbers.

stevenzwu · 2026-06-23T06:14:04Z

+   * <p>REPLACED files are the prior-state entries of v4 REPLACED/MODIFIED pairs and are not live.
+   * Returns null for manifest files written by pre-v4 writers.
+   */
+  default Integer replacedFilesCount() {


Generally, public-interface changes should be minimized and delayed. Justification for keeping these two on the interface in this PR:

In-tree consumers (this PR + close follow-ups):

TrackedFileAdapters.WrappedManifestInfo (this PR) forwards manifest.replacedFilesCount() / replacedRowsCount() when a v4 root manifest reference row is written.

MergingSnapshotProducer.validateAddedDVs (follow-up phase) filters concurrent data manifests by replacedFilesCount > 0 to detect v4 colocated-DV write conflicts — without it, every concurrent data manifest must be deep-scanned.

Convention. The interface-default pattern matches v3's existing extensions for spec-defined optional fields on ManifestFile: firstRowId (row lineage), keyMetadata (encryption), containsNaN (partition NaN). v4 spec-defined manifest_info counts belong on the same interface in the same shape.

A TrackedFile-based parallel v4 API was considered. Would require rewriting ManifestGroup and every engine that consumes FileScanTask. The cost is disproportionate to the savings.

Projected new default methods across the full v4 stack, all spec-defined additions backing v4 manifest_info / content_entry fields:

In this PR:

default Integer replacedFilesCount()

default Long replacedRowsCount()

default int formatVersion()

when root-level manifest DV bitmaps are implemented:

default ByteBuffer deletedPositions()

default ByteBuffer replacedPositions()

gaborkaszab

Hey @stevenzwu ,
I went through mostly for my own understanding. Left some comments, mostly I'm a bit hesitant to expose setting status and sequence numbers directly through the builders. Let me know what you think!

gaborkaszab · 2026-06-23T14:52:15Z

+
+  private ContentEntryAdapters() {}
+
+  static TrackedFile fromDataFile(


I don't see how the user will use these functions, but would it make sense to merge the fromDataFile and fromDeleteFile into a common fromManifestEntry? Then the user doesn't have to decide if it's data or delete, the adapter can do it instead.

I actually think we should keep them separate. The validation for delete files is different from data files and so is the error handling and messages, I think it's cleaner that way.

The callers also always know which type they have so it should be easier.

I agree with keeping these separate for data and delete file cases.

What I think we need to separate is ManifestEntry and DataFile. We accept DataFile instances through the API and create entires in operations like AppendFiles. The best use case is when accepting data files through the public API. There's a secondary use case for wrapping/adapting ManifestEntry for internal uses like rewriting v3 files to v4, but these are more narrow.

For now, I think we need the DataFile to TrackedFile adapter, not ManifestEntry.

gaborkaszab

Thanks @stevenzwu !
I did another pass today, but still didn't manage to drill down entirely.

gaborkaszab · 2026-07-20T10:49:52Z

 public interface ManifestFile {
  int PARTITION_SUMMARIES_ELEMENT_ID = 508;

+  /** Value of {@link #formatVersion()} for pre-v4 manifest files (v1, v2, and v3). */


Simply Format version for pre-v4 manifest files. ?

Will simplify to /** Format version for pre-v4 manifest files. */.

gaborkaszab · 2026-07-20T11:23:26Z

    return null;
  }

+  /** Returns the number of records in the manifest file, or null for pre-v4 manifests. */


Maybe emphasize this is not the sum of the referenced data files' record count, but the number of entries in the manifest file.
How about this?
/** Returns the number of entries in the manifest file, or null for pre-v4 manifests. */

Will update to Returns the number of entries in the manifest file, or null for pre-v4 manifests. — the field is the entry count of the manifest, not the sum of record counts of its data files.

gaborkaszab · 2026-07-20T11:24:40Z

  private Long firstRowId = null;
+  // v4+ root-manifest entry count; null for pre-v4 manifest list entries.
+  private Long recordCount = null;
+  // LEGACY_FORMAT_VERSION (0) for pre-v4 manifests; the table format version for v4+.


Since having similar comments for the new methods on the API, not sure if these comments are needed.

Will drop the field-level comment — the interface Javadoc on formatVersion() covers it.

gaborkaszab · 2026-07-20T11:25:38Z

  private PartitionFieldSummary[] partitions = null;
  private byte[] keyMetadata = null;
  private Long firstRowId = null;
+  // v4+ root-manifest entry count; null for pre-v4 manifest list entries.


Could you explain this v4+ root-manifest entry count?
I figured this is for leaf manifests too.

Wording is misleading — this field is the entry count of the manifest referenced by this manifest_list entry, applicable to any v4+ manifest (leaf or root in a split tree). Will drop the field-level comment; the interface Javadoc on recordCount() is authoritative.

gaborkaszab · 2026-07-20T11:38:15Z

+        deletedFilesCount,
+        deletedRowsCount,
+        firstRowId,
+        null /* recordCount */,


Instead of calling the other constructor with the default values spelled explicitly here, can't we just leave this variant of the constructor intact? With this implementation we spell the default values twice: Once when introducing the members, once here when passing params to the other constructor.

I am fine either way. but it seems that your described pattern seems more common in the existing code base. I will update.

gaborkaszab · 2026-07-20T13:34:00Z

+
+    @Override
+    void validateContent(DeleteFile newFile) {
+      // v4+ leaf delete manifests carry only equality deletes. DVs colocate on the data file's


I don't think we need to explain why we require the content to be eq-delete in a wrapper class for eq-deletes

Will remove — the class name and the check already convey it.

gaborkaszab · 2026-07-20T13:42:28Z

+      Preconditions.checkArgument(newManifest != null, "Invalid manifest file: null");
+      Preconditions.checkArgument(status != null, "Invalid status: null");
+      int formatVersion = newManifest.formatVersion();
+      Preconditions.checkArgument(


Hmm, opposed to ContentFiles, here we allow pre-v4 versions too. Is this intentional?

Intentional. A v4+ root manifest_list can reference either a v4+ manifest or a pre-v4 manifest that's still live (during migration, or when older manifests haven't been rewritten yet). Pre-v4 ManifestFile instances report LEGACY_FORMAT_VERSION (0) because the field is v4+ only. Both are accepted here.

gaborkaszab · 2026-07-20T13:44:07Z

+      int formatVersion = newManifest.formatVersion();
+      Preconditions.checkArgument(
+          formatVersion == ManifestFile.LEGACY_FORMAT_VERSION
+              || formatVersion >= TableMetadata.MIN_FORMAT_VERSION_ADAPTIVE_MANIFEST_TREE,


Do we guard against this being 1, 2 or 3 here? Would it make sense to silently use the legacy value for them?

Rejecting 1/2/3 is deliberate — those aren't valid values for the format_version field (v4+ only). A ManifestFile reporting 1/2/3 would signal a reader bug, not a legacy input; silently coercing would mask that. Pre-v4 inputs correctly report LEGACY_FORMAT_VERSION (0) and are covered by the first branch.

gaborkaszab · 2026-07-20T13:46:52Z

+              ? FileContent.DATA_MANIFEST
+              : FileContent.DELETE_MANIFEST;
+      this.recordCount = resolveRecordCount(newManifest);
+      this.tracking =


I thought that Tracking is an input for these adapters, even for manifests.

Content files (DataFile/DeleteFile) accept caller-built Tracking because status and sequence numbers depend on writer/commit-time context that lives outside the file. Manifest references have their tracking fully determined by the referenced manifest itself (snapshot_id, sequence_number, min_sequence_number), so wrap(...) builds it here. Open to taking Tracking as input for API consistency, but that shifts the reconstruction burden to callers with no new information.

gaborkaszab · 2026-07-20T13:59:12Z

+
+    @Override
+    public int replacedFilesCount() {
+      return zeroIfNull(manifest.replacedFilesCount());


Maybe I miss something, but it seems that manifest.replacedFilesCount() and manifest.replacedRowsCount() are always null.

Correct — GenericManifestFile doesn't override the interface defaults for replacedFilesCount/replacedRowsCount, so both resolve to null and zeroIfNull folds that to 0. Plumbing REPLACED aggregates from leaf-manifest writers up to the manifest_list entry is a follow-up; will add a TODO here so it's not forgotten.

Squashed rollup of v4_write_direction_wrappers for downstream cascading. Do not review here; see PR apache#16936 on the review branch for the multi-commit diff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

anoopj · 2026-07-24T21:01:58Z

+   * writer serializes this view directly through {@link StructLike}, so {@code copy} is not
+   * supported; a stable snapshot must be materialized via the writer instead.
+   */
+  static final class MapBackedContentStats implements ContentStats, StructLike, Serializable {


Optional: we generally add round trip serialization tests on serializable classes.

good catch. this class doesn't need to implement the serializable interface. I dropped it.

anoopj · 2026-07-24T21:17:31Z

  private PartitionFieldSummary[] partitions = null;
  private byte[] keyMetadata = null;
  private Long firstRowId = null;
+  private Long recordCount = null;


Looks like the CopyBuilder inner class may need to be updated to include these new fields?

Add reusable write-direction wrappers that present a legacy DataFile, DeleteFile, or ManifestFile as a v4 TrackedFile row for manifest serialization, re-pointed per row to avoid per-row allocation. Content stats are served by a reusable map-backed view over the file's stat maps, with copy() producing a stable snapshot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- ManifestFile: simplify LEGACY_FORMAT_VERSION and recordCount javadocs; clarify that recordCount is entries, not file records - GenericManifestFile: drop field-level comments (interface javadoc covers them); make the pre-v4 and v4+ constructors self-contained per the codebase convention (BaseSnapshot / GenericPartitionFieldSummary) instead of the pre-v4 delegating to the v4+ variant; drop stale arg-order-conflict note - TrackedFileAdapters: trim class javadoc to the one-line intent; drop redundant field-count and eq-delete validation comments; document the in-place re-point + fluent-return semantics on wrap(...); note that forDataFile / forEqualityDeleteFile accept pre-v4 source files (the formatVersion param gates the target manifest format only); invert sortOrderId so the base returns null and DataTrackedFile overrides; add TODO for wiring REPLACED aggregates on WrappedManifestInfo Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Move MapBackedContentStats out of TrackedFileAdapters into its own file, mirroring the top-level ContentStatsBackedMap on the read side (and its already top-level test). MapBackedFieldStats stays a private nested helper since it reads the parent's stat maps directly. Pure relocation with no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- Drop Serializable from MapBackedContentStats/MapBackedFieldStats. These reusable write-direction wrappers are never Java-serialized (the writer consumes them via StructLike, and their holders ContentTrackedFile / ManifestTrackedFile are not Serializable), so they don't need it and don't require a round-trip serialization test. - Carry recordCount and formatVersion through GenericManifestFile.CopyBuilder for non-GenericManifestFile inputs. The else branch built via the pre-v4 constructor and silently dropped both new v4 fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…eldStats apache/main (via apache#17322) added presence methods to FieldStats. Implement hasValueCount / hasNullValueCount / hasNanValueCount in MapBackedFieldStats, mirroring FieldStatsStruct. Also drop the redundant final class modifiers and make MapBackedFieldStats a non-static inner class, so it reads the enclosing stats view's maps directly instead of carrying an explicit parent reference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Squashed twin of stevenzwu/v4_write_direction_wrappers for use as a stable phase 1 base in the v4 read+write reset stack. Tree mirrors apache PR apache#16936 tip 259a3c4. Will be replaced by upstream when apache#16936 merges.

Squashed twin of stevenzwu/v4_write_direction_wrappers for use as a stable phase 1 base in the v4 read+write reset stack. Mirrors apache PR apache#16936 tip 259a3c4. Includes a compat fixup for stacking on top of PR apache#16958 (Phase 0), which renames TrackedFile.schemaWithContentStats(...): StructType to TrackedFile.schema(...): Schema. TrackedFileAdapters.TRACKED_FILE_FIELD_COUNT is updated to use the new name (with .asStruct() to keep field-counting semantics). Delta from PR apache#16936 alone is one line.

Production: - Implement the new FieldStats presence methods (hasValueCount / hasNullValueCount / hasNanValueCount) in MapBackedFieldStats. - Drop the -1 sentinel from the count getters; valueCount / nullValueCount / nanValueCount unbox directly and callers must check has*Count() first (matches FieldStatsStruct after apache#17322). - Drop redundant final modifiers and make MapBackedFieldStats a non-static inner class, reading the enclosing view's maps directly. Tests: - Assert upperBound type in testBoundDecodingPerType. - Add testSetNotSupported for the outer set(); split testFieldStatsCopyAndSetNotSupported into two focused tests. - Sharpen testContentStructLikeGetReturnsChildrenOrNull to verify the null slot maps to source field 5, not just that some position is null. - Split the StructLike surface out of testCountsOnlyColumnOmitsBounds into two symmetric tests (testDefaultStructLike / testCountsOnlyStructLike) using Comparators.forType with TestHelpers.Row expectations. - Extend testCountsAndDefaults to also cover the absent-count throw path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

TrackedFileAdapters: - Move final manifestInfo field to the top of ManifestTrackedFile so all final fields precede non-final ones. - Trim redundant Javadoc: drop "Instantiate once per writer / call wrap on each row" and "returns this for fluent usage" phrasing from the factory and wrap() methods, and tighten the @param docs (formatVersion "must be 4+"; drop "table schema for building X from the file's stats" style restatements while keeping the caller guidance on partitionType). TestMapBackedContentStats: - Add message checks on the auto-unbox NullPointerException assertions so checkstyle's AssertThatThrownByWithMessageCheck rule accepts them. TestTrackedFileAdapters: - Consolidate STATS_SCHEMA and TABLE_SCHEMA into one TABLE_SCHEMA (optional int id + optional float score) that backs both the read-side stat fixtures and the write-side wrapper factories. - Replace the local EMPTY_PARTITION_DATA constant with the shared PartitionData.EMPTY singleton. - Drop the V4_AND_ABOVE parameterization from testDataFileWrapperAdded; every other test in this file uses FORMAT_VERSION_V4 directly. - Move the assertNullTrackingFields / specsById / partition / dummyTrackedFile helpers to the end of the class so all private-static helpers live together after the tests. - Fill assertWriteDataFields coverage (sortOrderId, keyMetadata, manifestInfo/deletionVector/equalityIds nulls). - Fill testDataFileDoubleWrapRoundTrip DataFile-API coverage (content, partition via Comparators.forType, sortOrderId, splitOffsets, keyMetadata, firstRowId, nanValueCounts); document the two round-trip losses (columnSizes -> null, empty nan map -> null). - Reorder v4WriteManifestFile's last two params to match the trailing positions of the v4 GenericManifestFile constructor; add the missing status assertion in testManifestReferenceWrapperForV4Manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

stevenzwu · 2026-07-26T03:04:32Z

        null);
  }
-
-  private static DeletionVector deletionVector() {


this moved to a static variable near top.

@test

Added earlier in this PR for a parameterized test that was later converted to a plain @test; the constant now has no callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…avadoc The "reusable wrapper" phrasing in the summary sentence already conveys the one-per-writer usage; the trailing sentence just restates it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The class summary "Reusable view over a legacy ContentFile's stat maps" already conveys the reuse contract; drop the follow-up sentence that restated it and described the wrap() mechanism / perf rationale. Keep the lazy-decode note and the copy-unsupported guidance since those are load- bearing for callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The lazy-decode note and the copy-unsupported guidance describe internals of a package-private class. The copy() method itself throws with the "materialize via a writer instead" guidance, so callers who try it get the same message at runtime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-actions Bot added API core labels Jun 23, 2026

stevenzwu commented Jun 23, 2026

View reviewed changes

stevenzwu added this to V4: metadata tree Jun 23, 2026

gaborkaszab reviewed Jun 23, 2026

View reviewed changes

stevenzwu force-pushed the v4_write_direction_wrappers branch 5 times, most recently from c78d6b1 to 828e0c7 Compare June 25, 2026 21:35

github-actions Bot added docs build and removed API labels Jun 25, 2026

stevenzwu force-pushed the v4_write_direction_wrappers branch from 828e0c7 to fa86dc1 Compare June 25, 2026 22:18

github-actions Bot added the API label Jun 25, 2026

stevenzwu force-pushed the v4_write_direction_wrappers branch 5 times, most recently from fa46d3d to 55f5573 Compare June 26, 2026 05:57

stevenzwu mentioned this pull request Jun 26, 2026

Core: Adjustments for TrackedFileBuilder #16964

Closed

stevenzwu commented Jun 26, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/TrackedFileBuilder.java Outdated

stevenzwu force-pushed the v4_write_direction_wrappers branch 8 times, most recently from b5c92a0 to d5c696b Compare June 28, 2026 06:09

stevenzwu force-pushed the v4_write_direction_wrappers branch 9 times, most recently from cf615ba to bf2bb59 Compare July 20, 2026 05:23

stevenzwu marked this pull request as ready for review July 20, 2026 05:25

gaborkaszab reviewed Jul 20, 2026

View reviewed changes

stevenzwu force-pushed the v4_write_direction_wrappers branch from b3749a7 to e87edfb Compare July 23, 2026 18:32

anoopj reviewed Jul 24, 2026

View reviewed changes

stevenzwu and others added 5 commits July 24, 2026 15:08

stevenzwu force-pushed the v4_write_direction_wrappers branch from e87edfb to 259a3c4 Compare July 24, 2026 22:20

stevenzwu and others added 2 commits July 25, 2026 14:54

stevenzwu commented Jul 26, 2026

View reviewed changes

stevenzwu and others added 4 commits July 25, 2026 20:06

API: remove unused V4_AND_ABOVE test helper

862cb41

Added earlier in this PR for a parameterized test that was later converted to a plain @test; the constant now has no callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Core: drop redundant "Instantiate once per writer" from forDataFile J…

c66360f

…avadoc The "reusable wrapper" phrasing in the summary sentence already conveys the one-per-writer usage; the trailing sentence just restates it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>


		private ContentEntryAdapters() {}

		static TrackedFile fromDataFile(

Uh oh!

Conversation

stevenzwu commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Benchmark summary

Uh oh!

stevenzwu Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwu Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwu commented Jun 23, 2026 •

edited

Loading

stevenzwu Jun 23, 2026 •

edited

Loading

stevenzwu Jun 23, 2026 •

edited

Loading