Skip to content

fix(sync): land a hotfix staging already holds instead of failing on its conflict (ENG-4379) - #82

Merged
mindsdb-devops merged 2 commits into
mainfrom
fix/eng-4379-sync-hotfix-already-on-staging
Nov 17, 2026
Merged

mindsdb-devops merged 2 commits into
mainfrom
fix/eng-4379-sync-hotfix-already-on-staging

Conversation

@lucas-koontz

@lucas-koontz lucas-koontz commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

User story

As an engineer who ships a hotfix to production
I want my hotfix to reach staging on its own once it is on main
So that the next release merges cleanly and nobody gets paged about a fix staging already has

Why this matters

A hotfix lands on staging first and is then cherry-picked onto main. When staging has since edited a neighboring line, git merge main conflicts even though staging already holds the change, and the sync fails. On a squash-only staging nobody can land that merge by hand, so the branch stays red. cowork's sync failed this way on a one-line import.

What happens today

sync-main-to-staging.yml and release-unfreeze.yml run git merge --no-ff origin/main. Any conflict ends the step with git's own message, with no word on what staging lacks.

What should happen

Both call scripts/sync_staging.py. A clean merge is unchanged. On a conflict, it records the merge with -s ours, so staging's content stays and main becomes an ancestor of staging, only when two checks pass. First, every commit main gained since the merge base has an equal patch among the staging commits made since main last came in. Releases squash staging into main, so older staging commits are left out: one from releases ago can't vouch for a change staging has since undone. Second, main's changes that merge cleanly leave staging's tree as it is, which catches a change staging reverted. Otherwise the run fails, and its error names the conflicting files, the commits staging lacks and the files that differ. A landed merge names the held commits in its message.

Test plan

  • Held hotfix lands: a hotfix on staging, its cherry-pick on main and a later edit to the next line merge with staging's tree, main as an ancestor and the held commit in the merge message, with and without an earlier release. A plain merge conflicts on the same history.
  • Real case: replaying cowork's current main and staging in a local clone lands the merge with staging's tree unchanged and two parents.
  • Missing change still fails: one held and one missing commit fails, names the file and the missing commit, and leaves staging unchanged.
  • Not held: a main change whose only equal patch is a staging commit from two releases back fails, on a clean hunk and on a conflicting one. A change staging reverted fails and names the file, and so do a modify/delete and a delete/modify conflict.
  • Merge commit on main: fails.
  • Clean merge: unchanged.
  • Past failures: replaying the last 90 days of conflicting syncs across the release-train repos (22 runs) lands only cowork's, and 16 past clean syncs stay ordinary merges.
  • Each check fails its own tests when removed. pytest tests/, actionlint and workflow_graph.py pass.

@lucas-koontz
lucas-koontz requested a review from a team November 17, 2026 03:45
… staging's content only when main's clean changes are already there (ENG-4379)
@mindsdb-devops
mindsdb-devops merged commit 840e7c5 into main Nov 17, 2026
6 checks passed
@mindsdb-devops
mindsdb-devops deleted the fix/eng-4379-sync-hotfix-already-on-staging branch November 17, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants