Repository navigation
Checkpoint only fully applied binlog transactions - #1790
Open
coding-chimp wants to merge 2 commits into
Open
coding-chimp wants to merge 2 commits into
coding-chimp wants to merge 2 commits into
Conversation
This was referenced Oct 30, 2026
Route ordered completion markers through the existing DML queues and persist a separate applied-transaction watermark only after preceding SQL commits. Preserve batching and reject acknowledgements after failed delivery or migration cancellation. Make reader operations and checkpoint SQL cancellation-aware. Cover multi-row events, MySQL/MariaDB GTID cloning, partial-apply replay, failed batches, and cancellation during both checkpoint polling and an in-flight INSERT.
coding-chimp
force-pushed
the
bb/applied-transaction-checkpoint
branch
from
November 20, 2026 07:59
bb73bbb to
2f3c3b8
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved checkpointing defects can skip unapplied GTID rows after a crash and prevent recording post-rename recovery state on cancellation, causing resume to fail.
6 open issues
Keep cutover recovery writes independent of cancellation · New Persist applied transaction coordinates for cutover checkpoint · New Advance applied coordinates for filtered transactions · New Batch periodic checkpoint rows to avoid stale tables · New Release coordinate mutex before checkpoint writes · New Return stored failure when migration wait is canceled · New
What changed in this PR
Updates gh-ost’s binlog-to-applier pipeline so periodic resume checkpoints use completed transaction boundaries rather than row-level or heartbeat progress.
Changes:
- Pass ordered transaction-completion markers through the existing queues while preserving DML batching.
- Track an applied-transaction watermark and propagate cancellation through streaming and checkpoint writes.
- Clone GTID boundaries and add regression coverage for partial application, replay, failures, and cancellation.
| File | Description |
|---|---|
go/logic/transaction_marker_test.go |
Tests marker ordering, abort handling, GTID snapshots, and heartbeat safety. |
go/logic/streamer.go |
Dispatches completion markers and stops synchronous delivery on listener errors. |
go/logic/streamer_test.go |
Tests streaming shutdown after listener failure. |
go/logic/migrator.go |
Acknowledges committed transaction boundaries and updates checkpoint cancellation handling. |
go/logic/migrator_move_tables_cutover_test.go |
Tests move-tables checkpoint boundaries and marker-aware event delivery. |
go/logic/checkpoint_cancellation_test.go |
Tests cancellation during checkpoint waits, SQL writes, and background polling. |
go/logic/binlog_checkpoint_test.go |
Adds integration coverage for partial application, resume, batching, and failed SQL. |
go/logic/applier.go |
Stores the transaction watermark and supports cancellable checkpoint writes. |
go/logic/applier_test.go |
Verifies canceled writes preserve the durable checkpoint. |
go/binlog/transaction_test_helpers_test.go |
Adds shared helpers for synthetic transaction streams. |
go/binlog/transaction_marker_test.go |
Tests completion boundaries and cloned GTID coordinates. |
go/binlog/gomysql_reader.go |
Emits transaction markers and makes reads and queue sends cancellable. |
go/binlog/cancellation_test.go |
Tests cancellation of idle reads and blocked row or marker sends. |
go/binlog/binlog_entry.go |
Adds the transaction-completion flag to streamed entries. |
Review effort: Max (adaptive)
Use applied transaction boundaries for cutover recovery, keep recovery writes independent of migration cancellation, and batch checkpoint rows. Advance checkpoints for filtered transactions, avoid blocking apply during checkpoint writes, and return listener failures during startup.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Description
Fix a risk of checkpointing a partially applied binlog transaction. Multiple row changes can share the same binlog coordinates, so applying one does not prove that all have reached the ghost table. Resuming from such a checkpoint can skip unapplied changes.
script/cibuildreturns with no formatting errors, build errors or unit test errors.