Skip to content

Publish release binaries and the Axum image to GitHub Packages - #1543

Open
prk-Jr wants to merge 7 commits into
feature/binary-attestations-phase2from
feature/binary-attestations-packages
Open

prk-Jr wants to merge 7 commits into
feature/binary-attestations-phase2from
feature/binary-attestations-packages

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Why. A release publishes 19 assets, and each run adds 12 internal workflow artifacts, so operators can't tell which file to use. Each release now also appears under the repository's Packages as one package per thing to run:

    • trusted-server-cli, tagged <tag>-<target>;
    • trusted-server-fastly, tagged <tag>, with the wasm and the deployable trusted-server-fastly.tar.gz;
    • trusted-server-axum, a container image tagged <tag>.
  • CLI and Fastly packages are byte-for-byte copies of the release. Each file is one OCI layer whose digest is its SHA256SUMS entry, so they add no new trust. Users verify what they pull with the existing release verification.

  • The Axum image follows the image contract proposed in Make the Axum adapter a supported self-hosted deployment target #1236:

    • Debian slim, UID 10001, read-only root filesystem, no baked config or secrets;
    • binds through EDGEZERO__ADAPTER__HOST, so it listens on 0.0.0.0:8080;
    • a CI container smoke on every PR;
    • GitHub build provenance on the pushed digest.

    Axum stays development status.

  • Off by default. Both publishing jobs are skipped until a maintainer sets PACKAGE_PUBLICATION_ENABLED=true.

Stacked on #1449 (base feature/binary-attestations-phase2), merged up to fd33f2c74 (which includes main at 06e51be17). Phase 2's 27361ec58 added trusted-server-fastly.tar.gz to the release. Review it after #1449. When #1449 merges, this PR retargets to main.

Changes

File Change
.github/workflows/release.yml New packages job: checksum-pinned oras, contents: read and packages: write only. New axum-image job: build, push, digest check, actions/attest. Both need release and the new variable.
scripts/release/packages.mjs Publisher:
  • refuses drafts;
  • authenticates SHA256SUMS with gh release verify-asset;
  • requires the download to match it exactly;
  • refuses a release file that no package carries, so a new release asset stops the job instead of being left out;
  • counts a tag as absent only on the registry's <reference>: not found;
  • keeps tags that already hold exactly the release files, so a re-run completes a partial publish;
  • refuses any other existing tag;
  • checks each pushed manifest layer by layer.
scripts/release/packages.test.mjs Plan and mapping (including the Fastly .tar.gz layer), the exact SHA256SUMS check, refusal of a release file no package carries, the manifest check, first publish, re-run and partial completion, refusal of differing tags, fail-closed on registry errors, drafts, Dockerfile pins equal the repository pins.
scripts/release/phase1.test.mjs The publisher isolation test now reads each job on its own. It pins the packages job's sparse checkout, oras checksum and lack of release-write permission, and checks the axum-image gate.
crates/trusted-server-adapter-axum/Dockerfile Pinned by digest: Node from the .tool-versions image, Rust from the rust-toolchain.toml channel, and the Debian slim runtime. RUSTUP_TOOLCHAIN stops rustup downloads. Embeds the source commit as the version, uses STOPSIGNAL SIGINT, and runs as USER 10001:10001.
.dockerignore Excludes .git, build output, node_modules, local config and secrets, and docs from the build context.
crates/trusted-server-adapter-axum/src/main.rs Binds through edgezero_core::addr::resolve_bind_addr.
  • EDGEZERO__ADAPTER__HOST/PORT come first.
  • PORT is read only as the fallback, and PORT=0 is rejected.
  • The default stays 127.0.0.1:8787.
scripts/smoke-axum.sh AXUM_SMOKE_IMAGE runs every case against the image with a read-only root, no capabilities, and config and secrets passed by name. Adds a bridge-network case on the image's own defaults and an image-user check.
.github/workflows/test.yml New axum-image job: builds the image and runs the smoke against it.
.github/workflows/format.yml, AGENTS.md Run packages.test.mjs with the release script tests.
docs/guide/release-verification.md New "Download from GitHub Packages" section: what each package holds (the Fastly package includes the .tar.gz that "Deploy the verified Fastly package" deploys), verifying pulled files, and verifying the image by digest before running it. New maintainer checklist item: the variable, rehearsal first, visibility, recovery.
docs/guide/axum-dev.md The bind variables, and running the container image.
docs/superpowers/specs/2026-11-11-release-packages-design.md Design spec with two agent review rounds and a base-update note.

Closes

No issue closes. This is part of #1235 (attested releases). It also implements the image and bind items of #1236 ahead of #1236's spec. That spec may still change them, and the axum-image job is the one place to adjust. Publishing the image also takes on part of #897's scope early; the spec's scope note says so.

Spec

Spec: docs/superpowers/specs/2026-11-11-release-packages-design.md. It ends with the round 1 and round 2 review log.

Test plan

  • cargo fmt --all -- --check

  • cargo clippy-fastly && cargo clippy-axum && cargo clippy-cloudflare

  • cargo test-fastly && cargo test-axum && cargo test-cloudflare. Fastly was run with the pinned Viceroy 0.21.0.

  • JS tests: cd crates/trusted-server-js/lib && npx vitest run. 1,901 passed with the pinned Node 24.20.0. Local Node 24.12.0 fails 9 observe-npm tests that require exactly 24.20.0.

  • JS format: cd crates/trusted-server-js/lib && npm run format

  • Docs format: cd docs && npm run format

  • WASM build: not run separately. The Fastly adapter is unchanged.

  • Manual testing via fastly compute serve: not applicable.

  • Other:

    • node --test scripts/release/phase1.test.mjs scripts/release/packages.test.mjs: 54 passed.
    • shellcheck on the release and smoke scripts, actionlint, and the CI Node-pin guard.
    • Negative controls, each confirmed to fail when its protection is removed:
      • giving the packages job contents: write;
      • treating every registry error as "absent".
    • Local image (podman, arm64):
      • builds with no rustup downloads;
      • runs read-only as 10001:10001 and listens on 0.0.0.0:8080;
      • returns the expected 500 without config;
      • an invalid EDGEZERO__ADAPTER__HOST falls back to loopback with a warning;
      • stop takes about 0 seconds with SIGINT.
    • Fork rehearsal v0.0.0-rehearsal.37 (387f41b0b, before the alignment commit), the first publish to packages that did not exist yet:
      • all jobs passed, including packages and axum-image;
      • anonymous pulls work, and every layer digest matches SHA256SUMS;
      • the image runs as non-root with the right labels;
      • gh attestation verify passed for a CLI binary pulled from GHCR, and gh release verify-asset matched it to the release;
      • gh attestation verify passed for the image digest the tag points to.
  • This PR's Axum image smoke CI job, the first run of the container smoke on Linux host networking.

    • The image built.
    • The smoke ran every case against the container: the image-user check, the four missing-input cases, the positive origin request, and the bridge-network case on the image's own defaults.
    • It finished with "Axum first-success smoke passed in image trusted-server-axum:smoke".
  • Fork rehearsal v0.0.0-rehearsal.38 of dc1a063d8 (the last commit before the base merge, so its release had no .tar.gz): every job passed, including release, packages and axum-image.

    • All four packages hold files matching the release's SHA256SUMS, with the empty OCI config and a link to the source repository.
    • The image is linux/amd64 and runs as 10001:10001. It has stop signal SIGINT, sets EDGEZERO__ADAPTER__HOST=0.0.0.0 and EDGEZERO__ADAPTER__PORT=8080, and its labels show development status and revision dc1a063d8.
    • gh attestation verify passed for the image by the digest its tag points to.
  • After merging the base (27361ec58) and adding the .tar.gz layer:

    • node --test scripts/release/phase1.test.mjs scripts/release/packages.test.mjs: 56 passed with Node 24.20.0.
    • Negative controls: dropping the .tar.gz entry fails 2 tests; dropping the "no package carries" check fails 1.
    • cargo test-axum, cargo clippy-axum, cargo fmt --all -- --check, actionlint, the release shellcheck set and docs Prettier pass.
  • Fork rehearsal v0.0.0-rehearsal.41 of d719d2bfb (the .tar.gz fix plus the Phase 2 link fix): every job passed, including release, packages and axum-image. The first two attempts were cancelled only because no macOS arm64 runner was assigned.

  • PR CI on e5e6ea7fa: all 31 checks passed.

Checklist

  • Feature PRs: spec linked above, ending in a review log of two or more agent review rounds
  • Changes follow AGENTS.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses log macros (not println!). The Axum main keeps its existing eprintln! pattern for fatal startup errors, such as an invalid PORT.
  • New code has tests
  • No secrets or credentials committed

Closes #1235

Mirror each published release into GHCR: trusted-server-cli per target and
trusted-server-fastly as OCI artifacts that carry each binary with its SBOM
and Sigstore bundles, after checking every file against SHA256SUMS. Tags
are write-once and each pushed manifest is checked layer by layer.

Build the Axum server as a non-root distroless image from the tagged
source and attach GitHub build provenance to its digest. Axum gains
TRUSTED_SERVER_LISTEN_ADDR so a container can listen outside loopback.

Both jobs stay off until PACKAGE_PUBLICATION_ENABLED is true.
…e jobs

The publisher isolation test read everything after the release job, so the new packages and axum-image jobs made it count three checkouts. It now reads each job alone and pins the packages job's sparse checkout, pinned oras and lack of release-writing permission.
Run the image on Debian slim as UID 10001 with a read-only root, stop it
with SIGINT, and read the listen address from EdgeZero's
EDGEZERO__ADAPTER__HOST and EDGEZERO__ADAPTER__PORT instead of a new
variable. A test.yml job now builds the image on every PR and runs the Axum
smoke against it, including one case on the image's own bridge defaults.

Make package publication fail closed on registry errors, complete a partial
publish on re-run, authenticate SHA256SUMS first, and record two rounds of
spec review.
@prk-Jr prk-Jr self-assigned this Nov 11, 2026
@prk-Jr
prk-Jr marked this pull request as ready for review November 11, 2026 09:01
@aram356
aram356 added this pull request to stack #1492 November 11, 2026 17:25
Keep both new release-verification sections: the GitHub Packages download
guide and Phase 2's Fastly package deployment. Gate 16 in AGENTS.md lists
both the packages tests and the Fastly package validator, matching format.yml.
Phase 2 added trusted-server-fastly.tar.gz to the release and made it the
documented Fastly deploy path, but the trusted-server-fastly GHCR artifact
still carried only the wasm and its evidence. Add the archive as a gzip
layer; the wasm's provenance and SBOM bundles already cover it.

checkDownload now also refuses a release file that no package carries, so a
new release asset stops the packages job instead of being silently left out.
The tests built their release from the plan, so they could not see an
omission; a new case adds an unplanned file.
Bring in main's integration configuration validation (#1455) and mediator
fallback (#1493) through Phase 2.

@dhruv8sh dhruv8sh left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by a follow-up review with inline comments: #1543 (review)

@dhruv8sh dhruv8sh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Adds GHCR package publication for release binaries (byte-identical OCI artifacts checked against the authenticated SHA256SUMS) and an Axum container image with provenance, both gated behind PACKAGE_PUBLICATION_ENABLED. The publisher's fail-closed handling (exact-file check, not-found-only absence, manifest re-verification) is careful and well tested; only non-blocking observations below.

Non-blocking

🤔 thinking

  • Image stop signal path is not exercised by the smoke — see inline at scripts/smoke-axum.sh:160
  • PORT fallback duplicates EdgeZero's port-validity rule — see inline at crates/trusted-server-adapter-axum/src/main.rs:105

CI Status

  • cargo fmt: PASS (required)
  • cargo test: PASS (required)
  • format-typescript: PASS (required)
  • format-docs: PASS (required)
  • Axum image smoke: PASS
  • release scripts: PASS
  • vitest: PASS
  • cargo test (axum native): PASS
  • cargo test (cross-adapter parity): PASS
  • cargo check (cloudflare native + wasm32-unknown-unknown): PASS
  • cargo check/build/test (spin native + wasm32-wasip1): PASS
  • cargo test (ts CLI, native) (ubuntu-latest): PASS
  • cargo test (ts CLI, native) (macos-latest): PASS
  • integration tests: PASS
  • integration tests (Fastly EC lifecycle): PASS
  • browser integration tests: PASS
  • prepare integration artifacts: PASS
  • trace runtime and browser acceptance: PASS
  • HTML EOF runtime (Fastly): PASS
  • worker-deadlines (cloudflare): PASS
  • worker-deadlines (spin): PASS
  • docs-proposal scripts: PASS
  • CI Node pin guard: PASS
  • CLAUDE.md symlink guard: PASS
  • ts dev lint domains (changed lines): PASS
  • CodeQL: PASS
  • Analyze (rust): PASS
  • Analyze (actions): PASS
  • Analyze (python): PASS
  • Analyze (javascript-typescript): PASS

Comment thread scripts/smoke-axum.sh
Comment on lines +160 to +168
docker run --rm --init --pull never --name "$CONTAINER" \
--read-only --cap-drop ALL --security-opt no-new-privileges \
--publish "127.0.0.1:$DEFAULT_PORT:8080" \
"$IMAGE" >"$WORKSPACE/image-defaults.log" 2>&1 &
APP_PID=$!
smoke_wait_http "http://127.0.0.1:$DEFAULT_PORT/" 500 \
"$WORKSPACE/image-defaults.body" "$WORKSPACE/image-defaults.headers"
smoke_assert_process_alive "$APP_PID" "Axum image" "$WORKSPACE/image-defaults.log"
stop_app

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 thinking — The image's documented stop path is never exercised. Both image runs use --init (so tini, not the server, is PID 1) and end with stop_app, which runs docker rm --force (SIGKILL). STOPSIGNAL SIGINT and "docker stop shuts it down without waiting for the kill timeout" in docs/guide/axum-dev.md therefore have no CI coverage; a future change that drops the SIGINT handler or the STOPSIGNAL line would still pass this smoke.

Consider running this bridge-network case without --init (matching the documented docker run) and stopping it with docker stop, asserting it exits well before the timeout:

SECONDS=0
docker stop --time 10 "$CONTAINER" >/dev/null
CONTAINER=""
[ "$SECONDS" -lt 5 ] || smoke_die "image ignored its stop signal (took ${SECONDS}s)"
wait "$APP_PID" || true
APP_PID=""

Apply manually — not offered as a one-click suggestion because verifying it needs a full image build plus the container smoke, which this pass did not run.

edgezero_port: Option<&str>,
port: Option<&str>,
) -> Result<BindAddrResolution, String> {
let edgezero_port_valid =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 thinking — edgezero_port_valid re-implements resolve_port's validity rule from edgezero_core::addr (parses as u16 and is nonzero) to decide whether PORT is consulted. It matches EdgeZero v0.0.11 today, but if EdgeZero's rule changes (for example a range limit), this check and the real resolution would disagree, and an invalid PORT could stop startup even though EdgeZero would have used the EdgeZero port, or the reverse. A short comment naming the v0.0.11 coupling, or a test that asserts bind_address agrees with resolve_bind_addr on the zero and non-numeric cases, would make a future EdgeZero bump surface it.

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Adds opt-in GHCR publication of verified release binaries and an attested Axum container image, with bind-address configuration, container smoke coverage, and operator documentation. No actionable findings were identified across the 14 changed files.

Verification

All 31 reported CI checks pass. Passing suites were not rerun locally; package publication was not independently exercised.

CI Status

aram356 added a commit that referenced this pull request Nov 12, 2026
Per the user, the RC line is forward-ported to main (DEC-6). Rechecking the
proposed stack against DEC-3 and DEC-16: #1536 and #1537 are mostly
Axum-only stores, store selection and wiring that EZI-2b, T6 and T21
replace, so A2 becomes a new pull request that takes their tests, and both
close with #1535 once their code moves. The image and Terraform units
restack on #1543, which publishes the image and shares .dockerignore with
#1539. #1531 goes on #1506 for A1 and #1532 on #1531 for #1529.
aram356 added a commit that referenced this pull request Nov 12, 2026
Per the user, binary attestation and package publishing (#1449, #1463,
#1543) are not part of this work and carry lower priority (DEC-19). AX-5
keeps only the production image built from source, A6 shrinks to building
it and no longer waits on #1449, and the image and Terraform units restack
on #1531, where they apply cleanly, instead of #1543.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish attested release binaries with build provenance and SBOMs

3 participants