Track Iceberg v4 spec support #1107

This is the parent tracking issue for implementing the Iceberg format version 4 spec in iceberg-cpp.

iceberg-cpp currently supports up to v3 (TableMetadata::kSupportedTableFormatVersion = 3); #1070 adds the v4 table metadata foundations. Remaining v3 work is tracked separately in #959.

If you are interested in working on any of the items below, please open a dedicated sub-issue (or link an existing one) and reference this issue so progress is easy to follow. Where the Java implementation already exists, please keep behavior consistent with it; the relevant Java PRs are linked as references.

1. Foundation

• Accept format-version: 4 in table metadata parsing, creation and upgrade (bump kSupportedTableFormatVersion, add kMinFormatVersion* gates for v4-only features) (feat(metadata): add format v4 table metadata foundations #1070)
• v3 → v4 upgrade path and validation in TableMetadataBuilder::UpgradeFormatVersion (feat(metadata): add format v4 table metadata foundations #1070)
• Test fixtures: v4 table metadata and manifests produced by Java for interop tests

2. Relative paths

Spec: Paths in Metadata (apache/iceberg#13141, apache/iceberg#15630)

• Path resolution and relativization utilities (Java: Core: Add V4 location relativization utilities iceberg#16174)
• Make table metadata location optional in v4; must be absolute when present, otherwise supplied by the catalog (feat(metadata): add format v4 table metadata foundations #1070; Java: Core: v4 table metadata location should be optional iceberg#16572)
• Resolve relative locations when reading v4 metadata (manifests, deletion vectors, column files, statistics files, metadata log) (Java: Core: Resolve relative paths in V4 manifest reader iceberg#17434, Core: Preserve trailing slash on table location in V4 manifest reader iceberg#17807)
• Write relative locations by default for files under the table location
• Prepend a scheme to scheme-less locations from v1–v3 metadata for consistency with v4 absolute paths
• Catalog integration: pass the table location when loading metadata without location (REST: OpenAPI: IRC spec changes for v4 (relax responses in place) iceberg#17642)

3. Adaptive metadata tree

Spec: Manifests (apache/iceberg#16025)

3.1 Data model

• tracked_file, tracking, manifest_info, deletion_vector and column_file structs and their schemas (feat(metadata): add format v4 table metadata foundations #1070 adds FileTracking, DeletionVectorMetadata and ManifestInfo; full tracked_file and column_file are follow-ups; Java: API, Core: Introduce foundational types for V4 manifest support iceberg#15049, Core: Add TrackedFileStruct, implementation of v4 TrackedFile iceberg#15854, Core: Add builders for v4 structs iceberg#16092)
• New entry statuses REPLACED and MODIFIED, and content types DATA_MANIFEST / DELETE_MANIFEST
• Snapshot root-manifest field (replaces manifest-list in v4) in snapshot parsing and serialization (feat(metadata): add format v4 table metadata foundations #1070)

3.2 Read path

• Parquet manifest reader (v4 manifests are Parquet) (Java: Core: Add v4 manifest reader iceberg#16958)
• Read root manifests: inline data files, v4 leaf manifests and pre-v4 data/delete manifests
• Apply manifest deletion vectors (manifest_info.dv) and tracking.deleted_positions / replaced_positions
• Sequence number and first_row_id inheritance for v4 tracked files
• Adapters exposing v4 tracked files through the existing ManifestFile / DataFile / ManifestEntry abstractions (Java: API, Core: Add ManifestFile adapter for v4 tracked files iceberg#17932, API, Core: Read V4 manifests through ManifestFile iceberg#18021, API, Core: Expose co-located deletion vector through DataFile iceberg#17928)
• Scan planning with colocated deletion vectors (single-phase planning; no separate delete manifest pass for v4 leaf manifests)
• Continue to apply position and equality deletes from pre-v4 delete manifests in upgraded tables

3.3 Write path

• Parquet manifest writer for leaf and root manifests (Java: Core, Parquet: Allow for Writing Parquet/Avro Manifests in V4 iceberg#15634, Core: Add v4 TrackedFile write wrappers for leaf and root manifests iceberg#18853)
• SnapshotProducer / FastAppend / MergeAppend producing root manifests, including small commits that inline data files in the root manifest
• Manifest filter/merge managers for v4 (ManifestFilterManager, ManifestMergeManager, RewriteManifests)
• Row-level deletes written as deletion vectors colocated with the data file entry (MODIFIED + REPLACED entries)
• Reject new equality delete files and delete manifests in v4 tables
• Snapshot expiration and orphan file cleanup aware of root manifests, column files and colocated DVs
• Root manifest encryption key handling (pending Spec: Store root manifest encryption key directly in V4 snapshots iceberg#18503; depends on Support Iceberg table encryption (Java-compatible) #988)

4. Content stats

Spec: Content Stats (apache/iceberg#13153, apache/iceberg#14234)

• content_stats struct type derived from the table schema (stats field ID assignment: 10_000 + 200 * field-id, reserved ranges for _row_id and _last_updated_sequence_number)
• Per-field stats: lower_bound, upper_bound, tight_bounds, value_count, null_value_count, nan_value_count, avg_value_size_in_bytes
• geo_lower / geo_upper bounds for geometry and geography
• Convert between content stats and legacy metrics maps (Java: Core: Convert metrics to Content Stats iceberg#15251, Core: present v4 ContentStats as legacy stat maps via ContentStatsBackedMap iceberg#17322)
• Collect content stats in Parquet/Avro writers, including avg value sizes (Java: API, Core, Parquet: Carry avg value sizes for v4 content stats iceberg#17451)
• Evaluate content stats in InclusiveMetricsEvaluator / StrictMetricsEvaluator and manifest pruning (Java: Core: Add content stats evaluator iceberg#17413)
• File-level tight_stats (pending API, Core: Implement v4 file-level tight_stats iceberg#18598)
• Stats-only fields (stats-fields in table metadata), e.g. partition-value stats

5. Column updates

Spec: Column File (apache/iceberg#15146, apache/iceberg#16425)

• Track column_files on tracked files (metadata only)
• Read path: stitch column files with the base data file by row position
• Write path: commit column files as MODIFIED / REPLACED entries with correct sequence number handling

6. Schema and types

• file type: schema, JSON serde, Parquet/Avro/Arrow projection (File Type Support iceberg#17919)
• Default value expressions for write-default (Default Value Expressions iceberg#17616, Spec: Add support for default expression values iceberg#16777; depends on the expressions spec)
• Reject identifier-field-ids in v4 schemas (replaced by primary-key constraints)

7. Constraints

Spec: Constraints (apache/iceberg#17822)

• constraints and last-constraint-id in table metadata; constraint-statuses in snapshots
• Rewrite identifier-field-ids as an unenforced primary-key constraint when upgrading to v4
• Table update API and REST update/requirement types for constraints

8. Snapshots and commits

• Enforce strictly increasing timestamp-ms relative to the parent snapshot for commits to v4 tables
• Remove/disable the deprecated file-system (Hadoop-style) metadata commit scheme for v4 tables, if applicable

Still under discussion upstream (not yet in the spec)

These are tracked in the Iceberg V4 Spec milestone and will be added above once merged:

• vector type (Vector Type Support iceberg#18297, SPEC: Add vector type to v4 spec iceberg#18416)
• _last_updated_timestamp metadata column (Monotonic snapshot timestamp and expose a new metadata column _last_updated_timestamp_ms iceberg#17619, Spec: add _last_updated_timestamp metadata column to row lineage in v4 iceberg#18526)
• Collation support for string types (Proposal: Collation support for string types iceberg#17620)
• Derived columns (RFC: Derived column support in Iceberg spec iceberg#15923)
• V4 changelog planning (Core: Add V4 changelog planning iceberg#18852)

Suggested order

1. Foundation + relative paths (small, self-contained, already implemented in Java)
2. Tracked file data model + Parquet manifest reader + content stats read path → read-only v4 support
3. Manifest writers and commit path → v4 write support
4. Column updates, file type, constraints, default value expressions

If anything is missing or you have suggestions, feel free to comment or open additional sub-issues.