Repository navigation
Admin image delete: keep a platform manifest an index pushed meanwhile lists - #905
Merged
Merged
Conversation
Deleting a manifest locked the image with one SELECT ... FOR UPDATE and decided what goes with it on that statement's rows. The statement waits for a push that holds a platform manifest FOR SHARE, and its snapshot predates the index that push commits meanwhile: the delete took the platform manifest along, and the new tag's pull failed for that platform. The delete now reads the image a second time once it holds the locks and decides on that. A manifest only the second read returns arrived while the delete waited; it stays, and keeps what it lists and what refers to it. A delete of a manifest, or a batch of a whole image's delete, that deadlocks with a push (40P01) is run once more. Fixes #560 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJbTzcTACQUB2jX9EGh
schmitch
force-pushed
the
claude/560-image-delete-race
branch
from
November 17, 2026 09:41
96373e0 to
9df1a8b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #560
The bug
Deleting a manifest from the image page (
DELETE /admin/repositories/{name}/images?image=…&reference=sha256:…) locked the image with oneSELECT … FOR UPDATEand decided what goes with the manifest on that statement's rows. The statement waits for a push that holds a platform manifestFOR SHARE, and its snapshot predates the index that push commits meanwhile. The delete therefore took the platform manifest along, the new tag stayed, and pulling it failed for that platform.The fix
crates/server/src/admin/images.rs:delete_manifestreads the image a second time once it holds the locks (the issue's first proposal) and decides on that read. Whatever lists or tags a manifest the first statement locked had to commit before the lock was granted, so the second statement sees it.taken_alongkeeps it, with what it lists and what refers to it. The manifest asked for still goes.40P01) is retried once, for the manifest delete and for each batch of a whole image's delete. The transaction rolled back whole, as in the repository delete.Behaviour to be aware of: a referrer (signature, SBOM) pushed for the deleted manifest during that window stays, with its subject gone. I chose that over a lock-until-stable loop, which a steady stream of pushes could keep running.
The whole-image delete needed only the retry: it deletes every manifest, including one pushed meanwhile.
Left for other issues
set_tag,delete_tagandlock_image_for_retentionis unchanged. The retry here only covers the deadlock Admin image delete: a concurrent index push can lose a platform manifest a new tag still lists #560 names (id order against a push's digest order), and the tag delete has no retry.taken_alongtoo; it takes no lock, so it passes an empty set of arrivals and answers as before.No admin API or web UI change: the request and the answer are the same.
Verification
Run locally, with Postgres and versitygw from
mirror.gcr.iothrough testcontainers, and again after the rebase onto #902:cargo test -p forge-server --test registry: 55 passed. Two tests are new:an_admin_delete_keeps_a_platform_manifest_an_index_pushed_meanwhile_lists: a real push ofv2holds the platform manifestFOR SHAREand has inserted its index; the delete ofv1's index waits on that manifest; the push commits; the platform manifest must still be there.an_admin_delete_that_deadlocks_with_a_push_runs_again: a lock cycle with a transaction holding the manifests as a push does, for a manifest delete and a whole-image delete; the delete must answer 200.mainbefore the fix (checked before the rebase) and pass with it.cargo test -p forge-server --lib admin::images: 8 passed, one new (taken_alongkeeps what arrived).cargo xtask check(fmt and clippy-D warnings) andcargo xtask changelog --check: clean.Not run locally: the rest of the workspace's tests, the e2e suite and the
oci-e2escript; CI runs them.The deadlock test relies on Postgres aborting the delete, whose deadlock check runs first because it started waiting first. If Postgres aborted the test's transaction instead, the test would fail with a message saying so, not pass wrongly.
The changelog fragment is in
docs/changes/2.2.0/, and the issue moved fromBacklogto the2.2.0milestone, since it ships in the next release.🤖 Generated with Claude Code
https://claude.ai/code/session_012e6DJbTzcTACQUB2jX9EGh