[iceberg] Add metadata.iceberg.sync-full-history to rebuild full history - #9348
[iceberg] Add metadata.iceberg.sync-full-history to rebuild full history#9348vbabenkoru wants to merge 3 commits into
Conversation
When Iceberg metadata is created from scratch (compatibility enabled on a table that already has snapshots, or the previous metadata is unusable), only the latest Paimon snapshot was exposed to Iceberg, losing time travel and tags (apache#6107). With the new opt-in option the whole retained Paimon history is replayed instead: metadata is created afresh for the earliest retained snapshot, then every following snapshot is applied on top of its predecessor, so schemas, tags and the v3 row-id space accumulate exactly like live commits. Each step persists its metadata file, making an interrupted rebuild resumable; a resume base is first validated to cover the retained history prefix, so single-snapshot metadata from a plain rebuild never truncates the replayed history. Intermediate steps skip the version hint, the external catalog commit and cleanup, which only the final step performs. The rollback and self-heal floors (inherited table uuid, last-column-id and next-row-id) are threaded through every replay step so a rebuild never reuses ids handed out by abandoned metadata. Replayed snapshots stay subject to the snapshot retention policy exactly like live commits.
…scratch Creating metadata from scratch dropped every DataSplit that is not rawConvertible. For primary key tables this discards whole buckets whenever they contain level-0 files or overlapping key ranges, silently losing rows that the incremental commit path would have published (files above level 0 with their deletion vectors, or max-level files), and a full-history replay bakes the loss into every replayed snapshot. Collect files per file instead of per split: raw-convertible splits keep their exact export, and non-raw-convertible splits now contribute every file that shouldAddFileToIceberg accepts, together with its deletion vector. Files that genuinely cannot be read without merging (unmerged level-0 data) are counted and reported in a warning with exact file and row counts, pointing at full compaction as the immediate remedy.
|
A from-scratch full-history rebuild deleted the old build's manifest lists and manifests up front, so an external catalog still pointing at the old metadata could resolve files that no longer existed while the replay was running, or forever if it failed midway. The rebuild also could not recover from a corrupted metadata file: the resume scan tolerated it, but the cleanup walk parsed the same file unguarded and failed the commit. The up-front deletion is replaced by a tolerant collection of the old build's file names; each replay step removes only its own target file just before writing the replacement (a leftover from a regenerated build can match the step's commit identity while carrying other content), and the collected files are deleted only after the final step has published, skipping anything the replayed chain references. The expiration walk and the with-base read of a base metadata file now tolerate unreadable files as well, so a corrupted base self-heals like a structurally invalid one instead of failing every commit. Reported by JingsongLi in review.
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Inline review comments on full-history replay publication and recovery safety.
| Snapshot snapshot = snapshotManager.snapshot(snapshotId); | ||
| if (freshRebuild && snapshotId != currentSnapshotId) { | ||
| // see above; the final step replaces its twin through the regular write path | ||
| table.fileIO().deleteQuietly(pathFactory.toMetadataPath(snapshotId)); |
There was a problem hiding this comment.
[P1] Keep the published metadata file intact until replay commits
At this point v${snapshotId}.metadata.json can still be the file referenced by Hive's metadata_location or Hadoop's version hint, especially when snapshotId == currentSnapshotId - 1. Deleting it before createMetadataWithBase finishes creates a reader outage, and any later manifest/DV/metadata write failure leaves the external pointer aimed at a missing file until another callback succeeds. intermediate suppresses hint/catalog updates, but it does not make mutations to the already-published path invisible. Please build the replay under unique staging paths and switch the published head only after the final metadata is durable.
| if (isSameFormatVersion(metadata.formatVersion()) | ||
| && (formatVersion < IcebergMetadata.FORMAT_VERSION_V3 | ||
| || metadata.nextRowId() != null) | ||
| && coversRetainedPrefix(metadata, id, startId)) { |
There was a problem hiding this comment.
[P1] Reject unusable resume bases before selecting them
This candidate is not checked against the live snapshot identity or schema definitions. If an abandoned-timeline/schema-incompatible base triggered the rebuild, this loop selects the same base again; the final createMetadataWithBase rejects it and recursively enters rebuildFullHistory through recreateFromUnusableBase. I reproduced this with rollback, an Iceberg-disabled commit that reused snapshot 2, and a full-history commit for snapshot 3: it ends in StackOverflowError. Please apply all extension-usability checks here, or explicitly exclude the rejected base and restart a non-recursive fresh replay.
| "Unreadable base Iceberg metadata {}, recreating metadata.", | ||
| baseMetadataPath, | ||
| e); | ||
| recreateFromUnusableBase( |
There was a problem hiding this comment.
[P1] Do not reset v3 row lineage after an unreadable base
For a normal next commit, the caller floors are commonly zero. If the published v3 base is corrupt, this catch therefore rebuilds with a null UUID and nextRowIdFloor == 0, even though the unreadable metadata may already have issued higher row IDs. A regression test with v2.next-row-id == 3 produces snapshot 3 with first-row-id == 0, reusing IDs 0-2; before this change the commit failed instead of publishing reset lineage. For v3, recover the UUID/high-water mark from trustworthy older or catalog metadata, and fail safely when it cannot be recovered.
| boolean intermediate) | ||
| throws IOException { | ||
| if (intermediate) { | ||
| createMetadataWithoutBase( |
There was a problem hiding this comment.
[P1] Do not truncate history after an intermediate base-read failure
When intermediate is true, this fallback writes metadata containing only snapshotId. The replay loop then extends that truncated base and can publish the final file without revalidating coversRetainedPrefix, so one transient I/O error, corruption, or concurrent replacement silently drops all earlier retained snapshots and tags. Please propagate/retry the failure or restart from startId, and validate the retained prefix again before final publication.
2 participants
Purpose
When Iceberg metadata is created from scratch, only the latest Paimon snapshot was available in Iceberg. This happens when Iceberg compatibility is enabled on a table that already has snapshots, or when the old metadata can no longer be used. Time travel and tags were lost (#6107).
This PR has two commits.
1.
metadata.iceberg.sync-full-history(opt-in, defaultfalse): This option replays the full retained Paimon history. It creates new metadata for the earliest retained snapshot. It then applies each later snapshot on top of the previous one. This makes schemas, tags, and the row-id space for format version 3 build up in the same way as live commits.Design points:
inheritUuid,lastColumnIdFloor,nextRowIdFloor) are passed through every replay step. This prevents a rebuild from reusing row ids or column ids that abandoned metadata already assigned.snapshot.num-retained.*,snapshot.time-retained) works on replayed history in the same way as on live commits.2. Keep live-parity files when creating metadata from scratch. Creating metadata from scratch used to drop every
DataSplitthat is not raw-convertible. For primary key tables, this removed entire buckets when they contained level-0 files or overlapping key ranges. This silently lost rows that the incremental commit path would have published. A full-history replay then included that loss in every replayed snapshot. Files are now collected one file at a time instead of one split at a time. Raw-convertible splits keep their exact export. Non-raw-convertible splits add every file that the incremental path would accept, along with its deletion vector. Only unmerged level-0 data is excluded. These files are counted, and a warning recommends full compaction as the fix.Tests
IcebergSyncFullHistoryTest(paimon-core): checks that the default still exposes only the latest snapshot; checks a full replay of retained snapshots with schema changes and a tag in the middle of the history that an Iceberg client can read; checks that an interrupted replay resumes from the newest metadata; checks that a resume candidate without the retained history prefix is rejected; checks that changing the format version rebuilds history while keeping v3 row lineage correct.IcebergBootstrapNonRawSplitsTest(paimon-core): checks that creation from scratch exports compacted files from non-raw-convertible splits; checks full-history replay with non-raw splits.IcebergFullHistoryCompatibilityTest(paimon-iceberg): checks that enabling the option on an existing v3 DV table rebuilds history correctly with the Iceberg 1.8/1.11 readers; checks that an uncompacted DV bucket exports its compacted files.paimon-icebergtest suite passes on both configurations: JDK 11 / Iceberg 1.8.1 and JDK 17 /-Piceberg-ga(Iceberg 1.11).API and Format
Adds the optional table option
metadata.iceberg.sync-full-historywith a default offalse. The default behavior stays the same, except for the live-parity fix in commit 2. That fix only adds files that the incremental commit path would already publish.Documentation
The option is documented through its description in the generated configuration docs.
AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.