Skip to content

Admin image delete: keep a platform manifest an index pushed meanwhile lists - #905

Merged
schmitch merged 1 commit into
mainfrom
claude/560-image-delete-race
Nov 17, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/560-image-delete-race

Conversation

@schmitch

@schmitch schmitch commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Fixes #560

The bug

Deleting a manifest from the image page (DELETE /admin/repositories/{name}/images?image=…&reference=sha256:…) locked the image with one SELECT … FOR UPDATE and decided what goes with the manifest on that statement's rows. The statement waits for a push that holds a platform manifest FOR SHARE, and its snapshot predates the index that push commits meanwhile. The delete therefore took the platform manifest along, the new tag stayed, and pulling it failed for that platform.

The fix

crates/server/src/admin/images.rs:

  • delete_manifest reads the image a second time once it holds the locks (the issue's first proposal) and decides on that read. Whatever lists or tags a manifest the first statement locked had to commit before the lock was granted, so the second statement sees it.
  • A manifest only the second read returns arrived while the delete waited. What lists or tags it by now is not known, so taken_along keeps it, with what it lists and what refers to it. The manifest asked for still goes.
  • A deadlock (40P01) is retried once, for the manifest delete and for each batch of a whole image's delete. The transaction rolled back whole, as in the repository delete.

Behaviour to be aware of: a referrer (signature, SBOM) pushed for the deleted manifest during that window stays, with its subject gone. I chose that over a lock-until-stable loop, which a steady stream of pushes could keep running.

The whole-image delete needed only the retry: it deletes every manifest, including one pushed meanwhile.

Left for other issues

No admin API or web UI change: the request and the answer are the same.

Verification

Run locally, with Postgres and versitygw from mirror.gcr.io through testcontainers, and again after the rebase onto #902:

  • cargo test -p forge-server --test registry: 55 passed. Two tests are new:
    • an_admin_delete_keeps_a_platform_manifest_an_index_pushed_meanwhile_lists: a real push of v2 holds the platform manifest FOR SHARE and has inserted its index; the delete of v1's index waits on that manifest; the push commits; the platform manifest must still be there.
    • an_admin_delete_that_deadlocks_with_a_push_runs_again: a lock cycle with a transaction holding the manifests as a push does, for a manifest delete and a whole-image delete; the delete must answer 200.
  • Both new tests failed with the handler as it was on main before the fix (checked before the rebase) and pass with it.
  • cargo test -p forge-server --lib admin::images: 8 passed, one new (taken_along keeps what arrived).
  • cargo xtask check (fmt and clippy -D warnings) and cargo xtask changelog --check: clean.

Not run locally: the rest of the workspace's tests, the e2e suite and the oci-e2e script; CI runs them.

The deadlock test relies on Postgres aborting the delete, whose deadlock check runs first because it started waiting first. If Postgres aborted the test's transaction instead, the test would fail with a message saying so, not pass wrongly.

The changelog fragment is in docs/changes/2.2.0/, and the issue moved from Backlog to the 2.2.0 milestone, since it ships in the next release.

🤖 Generated with Claude Code

https://claude.ai/code/session_012e6DJbTzcTACQUB2jX9EGh

Deleting a manifest locked the image with one SELECT ... FOR UPDATE and
decided what goes with it on that statement's rows. The statement waits
for a push that holds a platform manifest FOR SHARE, and its snapshot
predates the index that push commits meanwhile: the delete took the
platform manifest along, and the new tag's pull failed for that platform.

The delete now reads the image a second time once it holds the locks and
decides on that. A manifest only the second read returns arrived while
the delete waited; it stays, and keeps what it lists and what refers to
it. A delete of a manifest, or a batch of a whole image's delete, that
deadlocks with a push (40P01) is run once more.

Fixes #560

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbTzcTACQUB2jX9EGh
@schmitch
schmitch force-pushed the claude/560-image-delete-race branch from 96373e0 to 9df1a8b Compare November 17, 2026 09:41
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit a976367 Nov 17, 2026
19 checks passed
@schmitch
schmitch deleted the claude/560-image-delete-race branch November 17, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Admin image delete: a concurrent index push can lose a platform manifest a new tag still lists

2 participants