Skip to content

Image page: read the newest of an image from the database, not all of it - #908

Queued
schmitch wants to merge 1 commit into
mainfrom
claude/561-image-page-bounded-query
Queued

schmitch wants to merge 1 commit into
mainfrom
claude/561-image-page-bounded-query

Conversation

@schmitch

Copy link
Copy Markdown
Member

Fixes #561

The problem

The image page (GET /browse/images/{repo}/{*image}) answers at most the newest 1,000 tags and 1,000 manifests (#240), but browse_images still loaded the whole image on every request: oci_dao::image_manifests had no LIMIT, aggregated every tag and ran a children subquery per manifest, and the server built an item and the digest map for every row before it truncated. A group did so once per serving member.

The change

  • oci_dao::image_listing replaces image_manifests (its only caller was the image page). Of one repository serving the image it returns the newest 1,000 tagged manifests with their tags (untagged_since IS NULL, which set_tag / delete_tag keep exact) and the newest 1,000 manifests. Everything else is read by key from those rows: the tags, an index's children, and the children's summaries (an index is sized by children the listing may have no room for, so the handler no longer needs every row for its digest map).
  • oci_dao::image_totals counts the image for tags_total and manifests_total.
  • Groups. Both take the serving members in priority order and a position. A lower priority member leaves out the tags a member before it has, and a digest a member before it holds comes as that member's manifest, so the handler's merge stays what it was: the first member's tag wins, a digest is listed once, both are counted once.
  • Migration 0020 adds oci_manifest_newest_idx (repository_id, image, created_at DESC, digest). No index knew a manifest's age, so "the newest N" could only be found by reading the image. The migration builds it in its transaction, which blocks pushes of manifests while it runs on a large table.
  • list_tags (the bounding and badging of Image detail page matches every tag against every manifest and pages nothing, so a large image blocks a runtime thread for seconds per request #240) is unchanged; it is now given the newest instead of everything.

What the page answers

  • Within the bounds: unchanged. I dumped the page's JSON before and after the change for a group of two members and each member alone (760 manifests and 758 tags in the group: tags both members have, digests both hold, several tags on a manifest, tag names that sort differently by locale, pushes at the same instant, indexes with children in either, both or neither member). The answers are identical.
  • Past the bounds, one repository: unchanged (the newest 1,000 tags by their manifest's push, their manifests, then the newest others).
  • Past the bounds, a group: one difference. Each member contributes the tags of its newest tagged manifests. When two members hold the same digest pushed at different times, the 1,000 listed tags can differ from the newest by the listed manifest's push. Choosing them exactly needs every tag of every member, which is what this issue removes. The totals, which digest a tag resolves to, and the manifest shown for a digest are exact. The doc comment on MAX_DETAIL_TAGS says so.

Query plans

PostgreSQL 18, one image of 50,000 manifests, each tagged, 4,000 child references, in a table of 180,000 manifests:

before after
rows returned 50,000 1,000
plan merge join of every manifest with every tag, GroupAggregate, a children subplan per manifest, sort two Index Scans on oci_manifest_newest_idx stopped by LIMIT, then nested loops by key (oci_tag_manifest_idx, oci_manifest_ref_pkey, the unique (repository_id, image, digest))
buffers about 150,000 about 9,500
execution 357 ms 10.6 ms
  • The totals are an Index Only Scan of the image's entries (no heap fetches), 28 ms for the 50,000. This part still grows with the image, as the issue's COUNT(*) does.
  • As a group's second member (30,000 manifests, 10,000 digests and 5,000 tags shared with the first): listing 25.6 ms with the same plan shape, totals 77 ms (hash anti joins over index-only scans).
  • A first draft joined the listed ids back to oci_manifest, and the planner hash-joined them to a sequential scan of the whole table. The lookups are lateral subqueries now, so they can only run once per listed row.

Tests

  • crates/db/tests/oci_image_page.rs (real Postgres): listing 1,000 of 30,000 manifests reads fewer than 10,000 oci_manifest rows and index entries and returns the newest with their tags; the totals; a lower priority repository lists and counts only what it adds, with an index's children read across both. It is a test binary of its own because the 30,000-manifest image changes the plans oci_images.rs counts rows of.
  • crates/server/tests/registry.rs: an image past the bounds over HTTP (totals, the newest 1,000 tags by name, their manifests newest first, an index sized by children that are not listed, an image with 3 old tags and 1,500 newer untagged manifests), and a group over two members past the bounds (shared tag, shared digests, totals, sources).

Run locally, all passing: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo xtask unused, cargo xtask doc-check, cargo xtask changelog --check, cargo xtask docs --check, cargo sqlx prepare --check --workspace, cargo test -p forge-db and cargo test -p forge-server (Postgres 18 and versitygw through mirror.gcr.io). After the rebase onto #902 I re-ran the lints, the sqlx check, forge-db and the server's registry binary, not the whole server suite.

Not run locally: the other crates' suites, forge-e2e, and the web checks (no file under web/ changes; the response's shape is the same).


Generated by Claude Code

The image page answers at most the newest 1,000 tags and 1,000 manifests
(#240), but browse_images still loaded the whole image on every request:
oci_dao::image_manifests had no LIMIT, aggregated every tag and ran a
children subquery per manifest, and the server built an item and the
digest map for every row before it truncated; a group did so once per
serving member.

oci_dao::image_listing reads, of each repository serving the image, its
newest 1,000 tagged manifests with their tags and its newest 1,000
manifests, through a new index (repository, image, push time), and the
rest by key from them: the children an index is sized by, and for a
group's lower priority member the tags and digests the members before it
hide. oci_dao::image_totals counts the image from its index entries for
tags_total and manifests_total. The page of an image within the bounds
is what it was.

Fixes #561

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJXrgzr2cNFuUgGtMFj
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Image page: the database still returns every manifest and tag of the image on each request

2 participants