Skip to content

git/gogit: add FetchAndReset and ErrPushRejected for push-conflict recovery - #1289

Open
monotek wants to merge 1 commit into
fluxcd:mainfrom
monotek:retry-push-conflict-fetch-reset
Open

git/gogit: add FetchAndReset and ErrPushRejected for push-conflict recovery#1289
monotek wants to merge 1 commit into
fluxcd:mainfrom
monotek:retry-push-conflict-fetch-reset

Conversation

@monotek

@monotek monotek commented Aug 31, 2026

Copy link
Copy Markdown

Why

In deployments with many independent ImageUpdateAutomation (IUA) objects pushing to the same branch of one GitOps repo (e.g. one IUA per service/region, all on independent reconcile timers), non-fast-forward push rejections become common once write concurrency is high enough. Today image-automation-controller has no in-process retry for a rejected push — the next attempt only happens on the next scheduled reconcile, or via controller-runtime's per-item exponential-backoff requeue (750ms doubling, capped at 15min). Under contention, a single IUA can lose several races in a row, each costing a full backoff cycle, turning a sub-minute git operation into a delay of tens of minutes.

A companion PR (fluxcd/image-automation-controller#1090) adds a retry-with-backoff loop there, analogous to a git pull --rebase && git push retry loop reusing the same local clone across attempts (no re-clone). go-git has no rebase API, so the equivalent here is: fetch (cheap, incremental) → hard-reset onto the new remote tip → let the caller recompute and recommit → push again. That retry loop needs two things this module doesn't currently expose:

  1. A way to tell a "lost the push race" error apart from any other push failure, so only genuinely recoverable errors get retried.
  2. A way to catch a working directory up to the new remote tip without a full re-clone (today gogit.Client only exposes Clone, not Fetch).

What

  • Adds git.ErrPushRejected, a sentinel wrapped onto Client.Push's returned error when the push is rejected because the remote branch moved ahead of the local repository's knowledge of it. go-git v5.19.1 has no usable exported sentinel for this on the Push path (ErrForceNeeded is fetch-only and unreachable from PushContext, confirmed by reading remote.go), so classification falls back to matching the "non-fast-forward" substring in go-git's own error text — documented in-code, with a regression test that fails loudly if a future go-git upgrade changes that wording.
  • Adds (*gogit.Client).FetchAndReset(ctx, branch): fetches branch from the remote via a scoped refspec and hard-resets the current worktree onto the fetched tip. Never touches the remote. Swallows go-git's NoErrAlreadyUpToDate. Mirrors SwitchBranch's plain (ctx, branchName) signature — no new config struct, since there's exactly one purpose and one caller today.
  • No changes to the repository.Reader/Writer/Client interfaces — image-automation-controller holds a concrete *gogit.Client, so this is purely additive new methods on the concrete type.

Testing

  • TestPush_NonFastForward_ReturnsErrPushRejected: two clients push from the same base; the loser's error satisfies errors.Is(err, git.ErrPushRejected).
  • TestFetchAndReset: a stale client with its own unpushed local commit recovers onto the winner's pushed tip, working tree content included; a second call with nothing new is a no-op.
  • make test-git passes, go vet/gofmt clean.

image-automation-controller can lose a push race when many independent
ImageUpdateAutomation objects push to the same branch. Recovering cheaply
requires distinguishing a rejected-due-to-conflict push from any other
push failure, and catching a working directory up to the new remote tip
without a full re-clone.

Add git.ErrPushRejected, wrapped onto Client.Push's error when go-git's
own pre-flight check rejects a non-fast-forward update (go-git has no
usable exported sentinel for this on the Push path). Add
FetchAndReset(ctx, branch), which fetches branch via a scoped refspec and
hard-resets the current worktree onto the fetched tip, swallowing
NoErrAlreadyUpToDate.

No changes to the Reader/Writer/Client interfaces; both are new methods
on the concrete gogit.Client type.

Assisted-by: Claude Sonnet 5/claude-sonnet-5
Signed-off-by: André Bauer <monotek23@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant