Skip to content

Checkpoint only fully applied binlog transactions - #1790

Open
coding-chimp wants to merge 2 commits into
github:masterfrom
Shopify:bb/applied-transaction-checkpoint
Open

coding-chimp wants to merge 2 commits into
github:masterfrom
Shopify:bb/applied-transaction-checkpoint

Conversation

@coding-chimp

Copy link
Copy Markdown
Contributor

Description

Fix a risk of checkpointing a partially applied binlog transaction. Multiple row changes can share the same binlog coordinates, so applying one does not prove that all have reached the ghost table. Resuming from such a checkpoint can skip unapplied changes.

  • Send ordered transaction-completion markers through the existing reader, dispatcher, and apply queues.
  • Advance a separate applied-transaction watermark only after preceding destination SQL commits. Persist that completed boundary rather than sampled reader or heartbeat progress.
  • Preserve bounded queues and DML batching across markers. Failed SQL batches cannot acknowledge collected markers.
  • Stop synchronous dispatch after listener errors and reject marker acknowledgement after abort. Make reader operations, checkpoint waits, and checkpoint SQL observe migration cancellation.
  • Clone MySQL and MariaDB GTID coordinates so later mutations cannot change recorded transaction boundaries.

In case this PR introduced Go code changes:

  • contributed code is using same conventions as original code
  • script/cibuild returns with no formatting errors, build errors or unit test errors.

Route ordered completion markers through the existing DML queues and persist a separate applied-transaction watermark only after preceding SQL commits. Preserve batching and reject acknowledgements after failed delivery or migration cancellation.

Make reader operations and checkpoint SQL cancellation-aware. Cover multi-row events, MySQL/MariaDB GTID cloning, partial-apply replay, failed batches, and cancellation during both checkpoint polling and an in-flight INSERT.
@coding-chimp
coding-chimp force-pushed the bb/applied-transaction-checkpoint branch from bb73bbb to 2f3c3b8 Compare November 20, 2026 07:59
Copilot AI adaptive review requested due to automatic review settings November 20, 2026 07:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved checkpointing defects can skip unapplied GTID rows after a crash and prevent recording post-rename recovery state on cancellation, causing resume to fail.

6 open issues
What changed in this PR

Updates gh-ost’s binlog-to-applier pipeline so periodic resume checkpoints use completed transaction boundaries rather than row-level or heartbeat progress.

Changes:

  • Pass ordered transaction-completion markers through the existing queues while preserving DML batching.
  • Track an applied-transaction watermark and propagate cancellation through streaming and checkpoint writes.
  • Clone GTID boundaries and add regression coverage for partial application, replay, failures, and cancellation.
File Description
go/​logic/​transaction_marker_test.go Tests marker ordering, abort handling, GTID snapshots, and heartbeat safety.
go/​logic/​streamer.go Dispatches completion markers and stops synchronous delivery on listener errors.
go/​logic/​streamer_test.go Tests streaming shutdown after listener failure.
go/​logic/​migrator.go Acknowledges committed transaction boundaries and updates checkpoint cancellation handling.
go/​logic/​migrator_move_tables_cutover_test.go Tests move-tables checkpoint boundaries and marker-aware event delivery.
go/​logic/​checkpoint_cancellation_test.go Tests cancellation during checkpoint waits, SQL writes, and background polling.
go/​logic/​binlog_checkpoint_test.go Adds integration coverage for partial application, resume, batching, and failed SQL.
go/​logic/​applier.go Stores the transaction watermark and supports cancellable checkpoint writes.
go/​logic/​applier_test.go Verifies canceled writes preserve the durable checkpoint.
go/​binlog/​transaction_test_helpers_test.go Adds shared helpers for synthetic transaction streams.
go/​binlog/​transaction_marker_test.go Tests completion boundaries and cloned GTID coordinates.
go/​binlog/​gomysql_reader.go Emits transaction markers and makes reads and queue sends cancellable.
go/​binlog/​cancellation_test.go Tests cancellation of idle reads and blocked row or marker sends.
go/​binlog/​binlog_entry.go Adds the transaction-completion flag to streamed entries.

Review effort: Max (adaptive)

Comment thread go/logic/applier.go Resolved
Comment thread go/logic/migrator.go Resolved
Comment thread go/binlog/gomysql_reader.go Resolved
Comment thread go/logic/applier.go Resolved
Comment thread go/logic/migrator.go Resolved
Comment thread go/logic/streamer.go Resolved
Use applied transaction boundaries for cutover recovery, keep recovery writes independent of migration cancellation, and batch checkpoint rows.

Advance checkpoints for filtered transactions, avoid blocking apply during checkpoint writes, and return listener failures during startup.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants