Backfill history.expire.max-ref-age-ms with Snapshot Expiration

[mkuchenbecker]

Summary

Use Iceberg's native reference expiration during snapshot-expiration maintenance.

New tables receive a seven-day history.expire.max-ref-age-ms default during creation. Snapshot expiration backfills the same property when it is missing from a legacy table, then delegates branch and tag retention to Iceberg. Existing table-level and reference-specific maximum ages remain authoritative.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

Added real Iceberg table tests that verify:

• Snapshot expiration backfills the seven-day maximum reference age for a legacy table.
• Native Iceberg expiration removes stale branch and tag references.
• A live branch remains when its snapshot is within the backfilled seven-day maximum reference age.
• A configured table-level maximum reference age is preserved and keeps live branch and tag references and their snapshots.
• New tables receive the seven-day default while an existing configured value is preserved.

The OperationsTest suite passes for the Iceberg 1.2 and Iceberg 1.5 application variants. The table repository tests and affected Spotless checks also pass.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

[cbb330]

Iceberg already executes this impl on the expireSnapshots commit.

The only real behavior change is our 7-day default vs iceberg's forever.

The extra removeBranch pass duplicates computeRetainedRefs, skips tags, and splits one atomic expire into two commits

Can we instead do this:

1. ensure all tables have max-ref-ages set in table properties on create
2. disable removing it
3. set guardrails on updating it
4. and then let iceberg's default snapshot expiration act on it?
5. backfill it to existing tables (or update the commit path to add it if it isn't there from create)

This would fix the ability for us to handle tags.

Also, there is a remaining TODO: which is for branches that commit often. they would not be expired using this flag. we need to expire those if they are missing a common ancestor with main (tying branch-forked-from-main-age to SE history policy)

[abhisheknath2011]

Iceberg already executes this impl on the expireSnapshots commit.

The only real behavior change is our 7-day default vs iceberg's forever.

The extra removeBranch pass duplicates computeRetainedRefs, skips tags, and splits one atomic expire into two commits

Can we instead do this:

1. ensure all tables have max-ref-ages set in table properties on create
2. disable removing it
3. set guardrails on updating it
4. and then let iceberg's default snapshot expiration act on it?
5. backfill it to existing tables (or update the commit path to add it if it isn't there from create)

This would fix the ability for us to handle tags.

Also, there is a remaining TODO: which is for branches that commit often. they would not be expired using this flag. we need to expire those if they are missing a common ancestor with main (tying branch-forked-from-main-age to SE history policy)

Thanks @cbb330 for the details!

@mkuchenbecker as the PR is updated now, can we update the PR description/headline as well before merging?