feat(commit): align functionality with java paimon commit by lucasfang · Pull Request #397 · alibaba/paimon-cpp

lucasfang · 2026-07-02T05:16:04Z

Purpose

Linked issue: close #xxx

Tests

API and Format

Documentation

Generative AI tooling

Copilot

Pull request overview

Note

Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR expands commit conflict detection and index-file handling in FileStoreCommitImpl, adds row-id/write-column conflict checking for data-evolution scenarios, and updates/extends unit tests accordingly.

Changes:

Add ConflictDetection, ManifestEntryChanges, and RowIdColumnConflictChecker to centralize commit change collection and conflict checks (including DV/global index considerations).
Support committing index files during overwrite / filter-and-overwrite flows, and adjust commit-kind selection (e.g., DV index append => overwrite kind).
Update commit retry behavior and add/adjust tests for index files, retries, and PK table commit behavior.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/paimon/core/operation/file_store_commit_test.cpp	Adds test validating DV index append triggers overwrite commit kind
src/paimon/core/operation/file_store_commit_impl_test.cpp	Extends commit tests for index files, retry metrics, and conflict checking signature changes
src/paimon/core/operation/file_store_commit_impl.h	Introduces new conflict/index APIs and wires in `ConflictDetection`
src/paimon/core/operation/file_store_commit_impl.cpp	Implements index-aware overwrite, retry commit checking, and centralized change collection
src/paimon/core/operation/file_store_commit.cpp	Removes prior PK-commit restriction logic in `Create`
src/paimon/core/operation/commit/row_id_column_conflict_checker.h	Adds row-id/write-column overlap conflict checker interface
src/paimon/core/operation/commit/row_id_column_conflict_checker.cpp	Implements row-id/write-column overlap conflict checker
src/paimon/core/operation/commit/manifest_entry_changes.h	Adds helper to collect manifest + index changes from commit messages
src/paimon/core/operation/commit/manifest_entry_changes.cpp	Implements change collection and “changed partitions” logic
src/paimon/core/operation/commit/manifest_entry_changes_test.cpp	Adds unit tests for `ManifestEntryChanges`
src/paimon/core/operation/commit/conflict_detection.h	Adds conflict detection API covering bucket/key-range/row-id/global-index checks
src/paimon/core/operation/commit/conflict_detection.cpp	Implements conflict detection logic and row-id history scan
src/paimon/core/operation/commit/conflict_detection_test.cpp	Adds unit test coverage for conflict detection behaviors
src/paimon/common/data/binary_row.h	Adds `std::hash` for tuple key used in conflict detection
src/paimon/CMakeLists.txt	Wires new sources + tests into the build

Comments suppressed due to low confidence (1)

src/paimon/core/operation/file_store_commit.cpp:1

The PR description is still the unfilled template (e.g., linked issue #xxx, tests section empty), but this change alters user-visible behavior by removing the previous PK-table commit restriction in FileStoreCommit::Create. Please update the PR title/description to clearly state the behavioral change, link the real issue, and list the tests validating the new PK commit behavior.

/*

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

zjw1111

Reviewed the commit conflict-detection alignment against Java Paimon. The port is largely faithful (ManifestEntryChanges collection order, RetryWaiter backoff math, SequenceSnapshotProperties ADD-only max, row-tracking assignment all match Java). A few correctness gaps and divergences are worth fixing before merge — see inline.

Highest priority: RowIdColumnConflictChecker::ConflictsWith swallows field-resolution errors and can let a schema mismatch pass conflict detection.

Also, a few items not inlined:

Test coverage: RowIdColumnConflictChecker has no C++ test at all (Java has 7 cases, incl. testUsesFieldIdAcrossRename / testFailsOnUnknownNonSystemWriteColumn which cover two of the issues below); CheckRowIdExistence / CheckRowIdRangeConflicts / SequenceSnapshotProperties also lack direct tests. Please add these.
Style (docs/code-style.md): plain int in row_id_column_conflict_checker.cpp binary search (low/high/mid/index) and std::unordered_set<int> in file_store_commit_impl.cpp NumChangedBuckets — use fixed-width types.
Intentional partial ports (please confirm and add a note): DV+UNAWARE entry enrichment, pkClusteringOverride short-circuit in CheckKeyRange, partition-expire handling in CheckDeleteInEntries, and dropStats on overwrite delete entries are all omitted.
The PR description is still the unfilled template — please fill in the change summary, linked issue, and tests.

Copilot

Pull request overview

Copilot reviewed 77 out of 77 changed files in this pull request and generated 2 comments.

zjw1111 · 2026-07-14T03:14:29Z

+        return field_name == SequenceNumber().Name() || field_name == ValueKind().Name() ||
+               field_name == RowKind().Name() || field_name == RowId().Name() ||
+               field_name == IndexScore().Name();


use IsSpecialFieldName?

zjw1111 · 2026-07-14T03:29:38Z

        /*commit_user=*/"commit_user_1", /*commit_identifier=*/9223372036854775807,
        /*commit_kind=*/Snapshot::CommitKind::Append(), /*time_millis=*/1758097357597,
-        /*log_offsets=*/std::map<int32_t, int64_t>(), /*total_record_count=*/5,
+        /*total_record_count=*/5,


zjw1111 · 2026-07-14T03:29:56Z

        /*commit_user=*/"commit_user_1", /*commit_identifier=*/9223372036854775807,
        /*commit_kind=*/Snapshot::CommitKind::Append(), /*time_millis=*/1758097357597,
-        /*log_offsets=*/std::map<int32_t, int64_t>(), /*total_record_count=*/5,
+        /*total_record_count=*/5,


zjw1111 · 2026-07-14T03:37:26Z


 class PAIMON_EXPORT CoreOptions {
 public:
+    // Specifies how to initialize the next sequence number for primary key table writers.


zjw1111

Cross-language review against Java Paimon found four additional correctness and boundary-condition gaps; see the inline comments.

The existing unresolved threads about overwrite ADD/DELETE deduplication and preserving the underlying commit error also still apply. I would not approve this revision until the P1 data-integrity issues are addressed.

zjw1111 · 2026-07-14T10:16:54Z

+                                              const std::vector<ManifestEntry>& delta_files,
+                                              std::vector<ManifestEntry>* snapshot_assigned) {
+    for (const auto& entry : delta_files) {
+        ManifestEntry assigned_entry = CloneEntryWithClonedFileMeta(entry);


ManifestEntry assigned_entry = entry only copies the shared DataFileMeta, so this assignment mutates the fixed provider's original input. If the atomic commit loses a CAS race, the next attempt receives metadata already stamped with the old snapshot ID and first-row ID; because those fields are now nonzero/non-null, it skips reassignment against the advanced latest snapshot and can commit overlapping row IDs or an old sequence. Java's entry.assignSequenceNumber / assignFirstRowId return new immutable entries. Please deep-copy the metadata (or make these APIs return new objects) and add a retry test where the first attempt fails and nextRowId advances concurrently.

zjw1111 · 2026-07-14T10:16:54Z

+        for (size_t j = group_start; j < i; ++j) {
+            const ManifestEntry& entry = entries_with_ranges[j].second;
+            if (IsDedicatedStorageFile(entry.FileName())) {
+                continue;


Skipping dedicated-storage files here omits Java's second half of the range check. Java verifies that every blob/vector row-ID range is fully contained by one data-file range; with this implementation, data [0,9] plus blob [5,14] returns OK. That can commit a dangling or cross-data-file mapping. Please split data and dedicated ranges and add the equivalent containment check.

zjw1111 · 2026-07-14T10:16:54Z

+
+    PAIMON_ASSIGN_OR_RAISE(Snapshot check_snapshot,
+                           snapshot_manager_->LoadSnapshot(row_id_check_from_snapshot_.value()));
+    if (!check_snapshot.NextRowId()) {


This should fail closed rather than silently disable history conflict detection. Java requires nextRowId to be present for rowIdCheckFromSnapshot and throws otherwise. Returning OK here lets migrated/older snapshots bypass the row-ID column conflict scan and may admit overlapping writes. Please return an invalid status that identifies the snapshot.

zjw1111 · 2026-07-14T10:16:54Z

+
+void RetryWaiter::RetryWait(int32_t retry_count) const {
+    int32_t non_negative_retry_count = std::max<int32_t>(0, retry_count);
+    double exponential = std::pow(2.0, static_cast<double>(non_negative_retry_count));


Clamp before converting to int64_t. With a sufficiently large configured commit.max-retries, min_retry_wait_ms_ * pow(2, retry_count) exceeds the integer range before std::min is applied; the conversion is not safe and can produce a non-positive delay, turning retries into a hot loop. Java applies Math.min(..., maxRetryWait) before the cast. A saturating calculation would preserve the configured maximum wait.

Copilot AI review requested due to automatic review settings July 2, 2026 05:16

Copilot AI reviewed Jul 2, 2026

View reviewed changes

lucasfang changed the title ~~Dev~~ feat(commit): align functionality with java paimon commit Jul 2, 2026

Copilot started reviewing on behalf of lucasfang July 2, 2026 11:43 View session

zjw1111 reviewed Jul 10, 2026

View reviewed changes

lucasfang force-pushed the dev branch from 8481ef3 to 2de05bd Compare July 14, 2026 09:45

feat(commit): align functionality with java paimon commit

f8c3ad2

lucasfang force-pushed the dev branch from 2de05bd to f8c3ad2 Compare July 14, 2026 09:48

lucasfang requested a review from Copilot July 14, 2026 09:51

Copilot started reviewing on behalf of lucasfang July 14, 2026 09:59 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/paimon/core/operation/metrics/commit_stats.h

Comment thread src/paimon/core/operation/commit/row_id_column_conflict_checker_test.cpp

zjw1111 reviewed Jul 14, 2026

View reviewed changes

Uh oh!

Conversation

lucasfang commented Jul 2, 2026

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjw1111 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants