Skip to content

Hybrid OCI/Docker manifest format breaks airgapped mirroring to Quay #778

Description

@butler54

Problem

The patterns-operator container images (patterns-operator, patterns-operator-console) are published as hybrid manifests: the parent is application/vnd.oci.image.index.v1+json (OCI format) but the children are application/vnd.docker.distribution.manifest.v2+json (Docker v2 format).

$ crane manifest quay.io/validatedpatterns/patterns-operator@sha256:eeb82d8c13fdb0c18603f11ee5cb16b8411806c1df3ebca75911fb2b87906306

mediaType: application/vnd.oci.image.index.v1+json   <-- OCI parent
manifests:
  - sha256:e6c2bbb5... type: application/vnd.docker.distribution.manifest.v2+json  <-- Docker child
  - sha256:c5de3adb... type: application/vnd.docker.distribution.manifest.v2+json  <-- Docker child

This hybrid format is rejected by Red Hat Quay (tested on v3.17.0 with FEATURE_GENERAL_OCI_SUPPORT: true, FEATURE_SPARSE_INDEX: true, FEATURE_HELM_OCI_SUPPORT: true all enabled) with MANIFEST_INVALID.

The same issue affects utility-container, imperative-container, and pattern-ui-catalog.

Impact

In airgapped/disconnected OpenShift deployments, oc-mirror v2 cannot mirror these images to Quay because:

  1. oc-mirror uses --preserve-digests by default (required for OpenShift digest-based image references)
  2. The containers/image library detects the hybrid format needs conversion (OCI parent → Docker v2 parent)
  3. Conversion would change the parent digest, violating --preserve-digests
  4. oc-mirror logs: Manifest list must be converted to type "application/vnd.docker.distribution.manifest.list.v2+json" to be written to destination, but we cannot modify it: "Instructed to preserve digests"

This forces workarounds (single-arch extraction, secondary mirror-registry) that add complexity to disconnected deployments.

Verification

Tested on Quay 3.17.0:

  • crane copy (verbatim push): MANIFEST_INVALID
  • oc image mirror --keep-manifest-list=true: manifest invalid
  • podman manifest push --all (creates new Docker v2 list): succeeds ✅ but parent digest changes
  • Pure Docker v2 manifest list images: push fine ✅
  • Pure OCI index images (OCI parent + OCI children): push fine ✅

Requested Fix

Publish all VP container images as either:

  • Pure OCI format (OCI index + OCI manifests), OR
  • Pure Docker v2 format (Docker manifest list + Docker v2 manifests)

Either format works with Quay. The hybrid format does not.

Environment

  • Quay v3.17.0
  • oc-mirror v2 (4.21.0-202607031221)
  • OpenShift 4.21.24
  • crane v0.20.x

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions