Skip to content

Publish verifiable build provenance for Server container images #792

New issue
New issue
Closed
Closed
Publish verifiable build provenance for Server container images#792
Labels
authority:githubGitHub is the authoritative lifecycle record for this workcompletion:evidence-requiredClose only after all explicit acceptance and operational evidence is publickind:defectA public product behavior is incorrectpriority:P3Lower-priority product improvementstatus:readyReady for implementation

Description

@rmcdaniel
rmcdaniel
opened on Oct 26, 2026

Problem

The currently published stable Server container images are reproducible across the two supported registries but have no cryptographic build-provenance attestation.

Read-only verification of 2.11.1 found:

  • durableworkflow/server:2.11.1 and ghcr.io/durable-workflow/server:2.11.1 resolve to the same amd64 and arm64 image digests;
  • each OCI index contains exactly two platform manifests and zero attestation manifests;
  • the current release workflow explicitly sets provenance: false and has no separate Cosign/GitHub artifact-attestation step.

The existing exact-tag, cross-registry digest, source, first-run, trust-store, Compose, and protocol checks remain valuable. The missing layer is a verifiable signed statement binding the published OCI digest to the repository and release workflow.

GitHub's current first-party guidance supports container attestations in public repositories on current plans with actions/attest@v4, the docker/build-push-action digest output, and push-to-registry: true. It specifically supports fully qualified Docker Hub (docker.io/...) and GHCR subject names.

Acceptance

  • Preserve the current single build digest and cross-registry per-platform digest equality.
  • Add only the required job permissions: id-token: write and attestations: write; retain the existing narrower permissions and protected dockerhub environment.
  • Use a full-SHA-pinned first-party actions/attest release to generate build provenance for the exact digest under both ghcr.io/durable-workflow/server and docker.io/durableworkflow/server, with the attestations pushed to the corresponding registries.
  • Do not enable BuildKit provenance if doing so would change the image index or break the existing cross-registry equality contract; a separate digest-bound attestation is sufficient.
  • Verify both registry references with gh attestation verify ... -R durable-workflow/server before promoting rolling aliases, and retain value-free attestation IDs/URLs in the existing release evidence.
  • Add focused workflow/source tests for permissions, exact subject names, digest binding, action SHA pinning, ordering, and fail-closed verification.
  • Qualify the first future published image end to end. Do not generate a misleading new-build attestation for an image produced by an older workflow.

This is P3 hardening of the existing stable release surface. It is not a gate on #325, Rust pre-RC development, or the current CI queue; implement it at a natural release-tooling checkpoint when CI has capacity.

Activity

  1. rmcdaniel
    added
    authority:githubGitHub is the authoritative lifecycle record for this work
    kind:defectA public product behavior is incorrect
    priority:P3Lower-priority product improvement
    status:readyReady for implementation
    completion:evidence-requiredClose only after all explicit acceptance and operational evidence is public
    on Oct 26, 2026

  2. rmcdaniel commented on Oct 26, 2026

    @rmcdaniel
    rmcdaniel
    on Oct 26, 2026
    MemberAuthor

    Published 2.11.3 follow-through confirms this issue's provenance gap on the new stable artifact; it does not reveal a release regression.

    • Release run 40414327142 completed successfully from exact source 8009904aac5d0a2bf082f06ec0f9f35b816d885a, and the 2.11.3 GitHub Release is public and non-prerelease.
    • Independent registry inspection shows both docker.io/durableworkflow/server:2.11.3 and ghcr.io/durable-workflow/server:2.11.3 as OCI indexes with exactly the same two image manifests: linux/amd64 sha256:c58d69b76ff30dc2511ad82b4e23c9f7d74545aba196b2d48ace06be66497fe1 and linux/arm64 sha256:65a6ee2bf9eaf53e0930c9e5f9c6238d35a69dce6a1e76131d8e495f2ac93b38. Neither index contains an attestation/unknown-platform manifest.
    • The exact committed release workflow still sets provenance: false, grants the publish job only contents: write and packages: write, and contains no id-token, attestations, or actions/attest step. Accordingly, no workflow path could have issued the GitHub digest-bound provenance required by this issue.
    • Positive release checks passed before completion: immutable tag/source checks, cross-registry publication, trust stores, bare first-run readiness, source-free Compose bootstrap, database TLS/certificate refusals, protocol-catalog convergence, cleanup, rolling-alias promotion, and anonymous pull/clean-client install of the matching immutable Helm chart.

    Disposition: 2.11.3 is qualified under the existing non-provenance controls, while #792 remains the P3 hardening owner for the next release-tooling checkpoint. This evidence does not justify rollback, republishing, a current release hold, or a Rust #325 gate.

  3. rmcdaniel commented on Oct 27, 2026

    @rmcdaniel
    rmcdaniel
    on Oct 27, 2026
    MemberAuthor

    Stable 2.11.4 follow-through supersedes the prior 2.11.3 observation and confirms the same bounded provenance gap without a release regression.

    • Annotated tag 2.11.4 resolves to commit ad3ac5fda0a30a58084b2d68569f2d91e23fabf3; exact-head release run 40447474983 completed successfully.
    • Fresh anonymous inspection shows both docker.io/durableworkflow/server:2.11.4 and ghcr.io/durable-workflow/server:2.11.4 contain exactly the same two image manifests: linux/amd64 sha256:bee30cde5b70a429561a49f25daf4ef71a1e405d43322f78ff1b8137029c80f3 and linux/arm64 sha256:cc297aa7d1f81b8bfa1169de6980dabe58d9776dcf513f379dcba2a51b1f5cc7. Their corresponding configuration digests and complete ordered layer-digest lists also match across registries.
    • Each index still contains only those two platform manifests and no attestation/unknown-platform manifest. The immutable 2.11.4 release workflow still sets provenance: false and has no separate actions/attest path, so this artifact has no GitHub digest-bound build-provenance attestation.
    • The release run passed the existing immutable source/dependency checks, cross-registry publication check, trust-store checks, bare first-run and source-free Compose checks, database-TLS refusal checks, protocol-catalog check, rolling-alias promotion, and clean anonymous install of OCI chart 0.3.4.

    Disposition remains unchanged: 2.11.4 is qualified under the existing controls, and #792 remains P3 hardening for a natural release-tooling checkpoint. This does not justify rollback, republishing, a release hold, or a Rust #325 gate.

  4. rmcdaniel commented on Oct 30, 2026

    @rmcdaniel
    rmcdaniel
    on Oct 30, 2026
    MemberAuthor

    Stable 2.11.7 follow-through supersedes the prior 2.11.4 observation and confirms the same bounded provenance gap without a release regression.

    • Annotated tag 2.11.7 resolves to exact commit f55f90e54a213872f1c164a1b22c082d859c18aa; normal release run 40919882321 completed successfully at that exact head.
    • Fresh anonymous registry reads show docker.io/durableworkflow/server:2.11.7 and ghcr.io/durable-workflow/server:2.11.7 resolve to the identical OCI index digest sha256:96e3e7de8c58f31bfb1ef4d5bddddc2678f4684f5cb3505e6e9ddc6fa3f7f625.
    • The complete index JSON is identical across registries. Each contains exactly linux/amd64 sha256:be7c2288ff935396262d743590530b204fac11b76582e56a525cc81a8da840d3 and linux/arm64 sha256:7dbc3db050fec05ea5cd10e07ac8399ecf3e285d593bca32979a2320f02a104f, with no unknown-platform or attestation manifest.
    • Current release source still sets BuildKit provenance: false and has no separate digest-bound attestation step. Existing exact-source, cross-registry, image, database-TLS, first-run, protocol and chart checks remain the release's accepted controls.

    Disposition is unchanged: 2.11.7 remains qualified under the existing controls; this issue owns P3 provenance hardening at a natural release-tooling checkpoint. Do not republish, roll back, hold current work, gate Rust #325 or create an extra release merely to exercise the future control.

  5. rmcdaniel commented on Oct 30, 2026

    @rmcdaniel
    rmcdaniel
    on Oct 30, 2026
    MemberAuthor

    Fresh anonymous rolling-alias drift readback (2026-10-30 02:20 UTC):

    • docker.io/durableworkflow/server:{2.11.7,2.11,2,latest} all resolve to exactly linux/amd64 sha256:be7c2288ff935396262d743590530b204fac11b76582e56a525cc81a8da840d3 and linux/arm64 sha256:7dbc3db050fec05ea5cd10e07ac8399ecf3e285d593bca32979a2320f02a104f.
    • ghcr.io/durable-workflow/server:{2.11.7,2.11,2,latest} resolve to the same two platform manifests.
    • No inspected reference contains an extra platform/attestation entry.

    The immutable version and all customer-facing rolling aliases therefore remain aligned across both registries after the subsequent main merge. #792's P3 provenance gap and prior no-hold disposition are unchanged. No image, tag, registry, release, credential or production state changed during this readback.

  6. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    rmcdaniel
    on Nov 4, 2026
    MemberAuthor

    Server open-item pass, 2026-11-04.

    The native release line now delivers this provenance requirement. The exact published-index preflight, artifact acceptance and accepted image run41805416467 bind F, index651253bd, both platform manifests, both registry names and the workflow/source/tag-bound build attestations. The normal native release helper verifies the subjects before later release/promotion operations. This is separate digest-bound provenance, not an inferred BuildKit index entry.

    I reconciled the older PHP-only observations with the current native implementation and actual immutable artifact evidence. No attestation is fabricated for an old PHP build. SBOM generation remains open in #1012, vulnerability scanning in #802, and alias qualification ordering in #1020. Closing only the delivered native provenance item; the frozen 3.1 GitHub Release and full failover/recovery holds remain unchanged.

  7. rmcdaniel
    closed this as completedon Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    authority:githubGitHub is the authoritative lifecycle record for this workcompletion:evidence-requiredClose only after all explicit acceptance and operational evidence is publickind:defectA public product behavior is incorrectpriority:P3Lower-priority product improvementstatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions