From 918de6a1aab47b0f26cad603f741064cfe747c24 Mon Sep 17 00:00:00 2001 From: Toby Martin Date: Fri, 21 Aug 2026 08:31:48 +1000 Subject: [PATCH] ci: split release validation into a job matrix The release workflow ran every lane serially in one `validate` job, so one runner linked every package, backend, and provider permutation into a single `target/`. Disabling debug info bought headroom but left the structural cause: `scripts/check-ai-provider-lanes.sh` alone relinks the whole graphql-orm-ai test binary set seven times in that directory. Decompose release validation the way CI already is - workspace integrity, neutral packages, ORM, alternate features, AI GraphQL naming, AI PostgreSQL parity, router, router MSRV - and fan the AI provider lanes across a matrix so each provider feature gets its own runner and its own `target/`. The lane set comes from `--list` in the lane runner rather than being restated in YAML, so a new provider cannot be silently dropped from the release matrix. Every command the single job ran still runs, with the lockfile fixed. The one exception is the catalogue/tool-profiles rustdoc lane, which the job ran twice with identical flags and now runs once inside the superset neutral-package lane. A `validate` fan-in job requires every lane to have succeeded, and asserts it explicitly so a future condition on a lane cannot quietly narrow what an immutable release was verified against. Tagging, attestation, and publication are unchanged and depend on that fan-in. The identity proof keeps the protected `release` environment on the workflow's entry job, so one human approval still gates the run before any lane compiles anything, and the publication job depends on it. The disk reclaim step becomes a local composite action and is used where compiling still fills a runner: the provider lanes and the optional router artifact build. Documentation impact: documentation updated. The release runbook now states that validation is parallel jobs that each own their `target/`, and the testing guide documents the `--list` lane enumeration. Co-Authored-By: Claude Opus 5 (1M context) --- .../actions/reclaim-runner-disk/action.yml | 25 ++ .github/workflows/release.yml | 296 +++++++++++++++--- docs/development/testing.md | 6 +- docs/operations/release/process.md | 7 +- scripts/check-ai-provider-lanes.sh | 26 +- 5 files changed, 312 insertions(+), 48 deletions(-) create mode 100644 .github/actions/reclaim-runner-disk/action.yml diff --git a/.github/actions/reclaim-runner-disk/action.yml b/.github/actions/reclaim-runner-disk/action.yml new file mode 100644 index 00000000..06bde88f --- /dev/null +++ b/.github/actions/reclaim-runner-disk/action.yml @@ -0,0 +1,25 @@ +name: Reclaim runner disk +description: >- + Drop preinstalled toolchains this workspace never builds against, so a + compile-heavy lane cannot exhaust the GitHub-hosted runner image while + linking. Reclaiming disk never drops, reorders, or weakens a verification + lane. + +runs: + using: composite + steps: + - shell: bash + run: | + set -euo pipefail + df -h / + sudo rm -rf \ + /usr/local/lib/android \ + /usr/share/dotnet \ + /opt/ghc \ + /usr/local/share/powershell \ + /usr/local/share/boost \ + /opt/hostedtoolcache/CodeQL \ + "${AGENT_TOOLSDIRECTORY:-/opt/hostedtoolcache}/Python" \ + "${AGENT_TOOLSDIRECTORY:-/opt/hostedtoolcache}/node" || true + sudo docker image prune --all --force || true + df -h / diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 02fd3c19..3ac30a67 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -35,22 +35,28 @@ concurrency: group: workspace-release cancel-in-progress: false -# Every release lane shares one target directory in one job, across all -# packages and every backend and provider permutation. Debug info and -# incremental artifacts dominate that directory and contribute nothing to a -# test, Clippy, rustdoc, or check lane, so they are disabled rather than -# dropping or weakening any lane. Failure backtraces lose line detail; the -# split CI workflow keeps full debug info for diagnosis. +# Release validation is decomposed into independent jobs that mirror the CI +# workflow, and the AI provider lanes fan out one runner per provider feature, +# so no single `target/` ever holds every package, backend, and provider +# permutation. Debug info and incremental artifacts still dominate whatever a +# lane leaves behind and contribute nothing to a test, Clippy, rustdoc, or +# check lane, so they stay disabled rather than dropping or weakening any lane. +# Failure backtraces lose line detail; the CI workflow keeps `line-tables-only` +# for diagnosis. env: CARGO_PROFILE_DEV_DEBUG: "0" CARGO_PROFILE_TEST_DEBUG: "0" CARGO_INCREMENTAL: "0" jobs: - validate: - name: Validate exact release source + guard: + name: Prove the selected source is current main runs-on: ubuntu-latest - timeout-minutes: 90 + timeout-minutes: 10 + # The protected `release` environment stays on the single entry job. Every + # validation lane and the publication job depend on it, so one human + # approval still gates the whole run before any release lane compiles + # anything, exactly as the single `validate` job required. environment: release steps: - uses: actions/checkout@v5 @@ -76,30 +82,20 @@ jobs: echo "router_distribution_approval is required for a router binary" >&2 exit 1 fi - - name: Reclaim runner disk - run: | - set -euo pipefail - # Every release lane runs in this one job and shares one target - # directory: all packages, every backend and provider permutation, - # rustdoc output, and a second toolchain. CI splits the same lanes - # across jobs, so only the release job exhausts the runner image. - # Drop preinstalled toolchains this workspace never builds against - # rather than dropping or reordering any verification lane. - df -h / - sudo rm -rf \ - /usr/local/lib/android \ - /usr/share/dotnet \ - /opt/ghc \ - /usr/local/share/powershell \ - /usr/local/share/boost \ - /opt/hostedtoolcache/CodeQL \ - "${AGENT_TOOLSDIRECTORY:-/opt/hostedtoolcache}/Python" \ - "${AGENT_TOOLSDIRECTORY:-/opt/hostedtoolcache}/node" || true - sudo docker image prune --all --force || true - df -h / + + workspace-integrity: + name: Workspace integrity and release metadata + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + fetch-depth: 0 - uses: dtolnay/rust-toolchain@stable with: - components: clippy,rustfmt + components: rustfmt - uses: Swatinem/rust-cache@v2 - name: Validate documentation and release metadata run: | @@ -109,6 +105,20 @@ jobs: scripts/check-release-manifest.sh scripts/check-workspace-dependencies.sh cargo fmt --all -- --check + + neutral-packages: + name: Neutral package lanes + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - uses: dtolnay/rust-toolchain@stable + with: + components: clippy + - uses: Swatinem/rust-cache@v2 - name: Verify neutral packages run: | cargo test -p graphql-orm-operation-catalog -p graphql-orm-ai-tool-profiles --locked @@ -118,6 +128,20 @@ jobs: cargo clippy -p graphql-orm-backup --all-targets --features orm-sqlite --locked -- -D warnings RUSTDOCFLAGS="-D warnings -D missing_docs" cargo doc -p graphql-orm-operation-catalog -p graphql-orm-ai-tool-profiles -p graphql-orm-storage --no-deps --locked RUSTDOCFLAGS="-D warnings" cargo doc -p graphql-orm-backup --features orm-sqlite --no-deps --locked + + orm: + name: ORM backend lanes + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - uses: dtolnay/rust-toolchain@stable + with: + components: clippy + - uses: Swatinem/rust-cache@v2 - name: Verify ORM backend lanes run: | cargo test -p graphql-orm --locked @@ -125,22 +149,126 @@ jobs: cargo check -p graphql-orm --no-default-features --features mssql --locked cargo clippy -p graphql-orm --all-targets --locked -- -D warnings RUSTDOCFLAGS="-D warnings" cargo doc -p graphql-orm -p graphql-orm-macros --no-deps --locked - - name: Verify AI provider and backend lanes + + ai-provider-matrix: + name: Enumerate AI provider lanes + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 10 + outputs: + providers: ${{ steps.providers.outputs.providers }} + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - id: providers + name: Read the provider lane list from the lane runner + # The provider list stays owned by scripts/check-ai-provider-lanes.sh. + # Reading it here means a provider added to the script cannot be + # silently dropped from the release matrix. + run: | + set -euo pipefail + providers=$(scripts/check-ai-provider-lanes.sh --list) + echo "${providers}" + printf 'providers=%s\n' "${providers}" >> "${GITHUB_OUTPUT}" + + ai-provider-lanes: + name: AI provider lane (${{ matrix.provider }}) + needs: + - guard + - ai-provider-matrix + runs-on: ubuntu-latest + timeout-minutes: 60 + strategy: + # Release runs are rare and gate an immutable publication, so report + # every failing provider lane instead of cancelling the siblings. + fail-fast: false + matrix: + provider: ${{ fromJSON(needs.ai-provider-matrix.outputs.providers) }} + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - name: Reclaim runner disk + # graphql-orm-ai relinks every test binary for the test, Clippy, and + # rustdoc lane of this provider. One provider per runner is what fixes + # the exhaustion; reclaiming the unused preinstalled toolchains keeps + # headroom for the crate that originally filled the disk. + uses: ./.github/actions/reclaim-runner-disk + - uses: dtolnay/rust-toolchain@stable + with: + components: clippy + - uses: Swatinem/rust-cache@v2 + with: + key: ${{ matrix.provider }} + - name: Verify the provider lane + env: + PROVIDER: ${{ matrix.provider }} + run: | + scripts/check-ai-provider-lanes.sh test "${PROVIDER}" + scripts/check-ai-provider-lanes.sh clippy "${PROVIDER}" + scripts/check-ai-provider-lanes.sh doc "${PROVIDER}" + + ai-graphql-naming: + name: AI GraphQL naming lane + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + - run: cargo test -p graphql-orm-ai --features graphql-case-pascal --test graphql_naming --locked + + ai-postgres-parity: + name: AI PostgreSQL parity lane + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + - name: Run ORM parity against a test-owned disposable PostgreSQL container + run: cargo test -p graphql-orm-ai --no-default-features --features postgres,provider-openai --test postgres_parity --locked -- --test-threads=1 + + alternate-features: + name: Alternate feature lanes + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + - name: Verify alternate AI, storage, and backup lanes run: | - scripts/check-ai-provider-lanes.sh test - cargo test -p graphql-orm-ai --features graphql-case-pascal --test graphql_naming --locked cargo check -p graphql-orm-ai --no-default-features --features postgres --locked cargo check -p graphql-orm-ai --no-default-features --features mssql --locked cargo check -p graphql-orm-ai -p graphql-orm --no-default-features --features graphql-orm-ai/sqlite,graphql-orm/mssql --locked - cargo test -p graphql-orm-ai --no-default-features --features postgres,provider-openai --test postgres_parity --locked -- --test-threads=1 - scripts/check-ai-provider-lanes.sh clippy - scripts/check-ai-provider-lanes.sh doc - RUSTDOCFLAGS="-D warnings -D missing_docs" cargo doc -p graphql-orm-operation-catalog -p graphql-orm-ai-tool-profiles --no-deps --locked - - name: Verify alternate storage and backup lanes - run: | cargo check -p graphql-orm-backup --no-default-features --features local,orm-postgres --locked cargo check -p graphql-orm-storage --no-default-features --features s3,azure --locked cargo check -p graphql-orm-storage --no-default-features --features smb --locked + + router: + name: Router lanes + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - uses: dtolnay/rust-toolchain@stable + with: + components: clippy + - uses: Swatinem/rust-cache@v2 - name: Verify router lanes run: | cargo test -p graphql-orm-router-protocol --locked @@ -150,11 +278,93 @@ jobs: cargo clippy -p graphql-orm-router --all-targets --features auth-agql --locked -- -D warnings RUSTDOCFLAGS="-D warnings -D missing_docs" cargo doc -p graphql-orm-router-protocol --no-deps --locked RUSTDOCFLAGS="-D warnings" cargo doc -p graphql-orm-router --no-deps --features auth-agql --locked + + router-msrv: + name: Router minimum supported Rust version + needs: guard + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + - uses: dtolnay/rust-toolchain@master + with: + toolchain: 1.90.0 + - uses: Swatinem/rust-cache@v2 - name: Verify router minimum supported Rust version run: | - rustup toolchain install 1.90.0 --profile minimal - cargo +1.90.0 test -p graphql-orm-router-protocol --locked - cargo +1.90.0 test -p graphql-orm-router --locked + cargo test -p graphql-orm-router-protocol --locked + cargo test -p graphql-orm-router --locked + + validate: + name: Release validation complete + needs: + - guard + - workspace-integrity + - neutral-packages + - orm + - ai-provider-matrix + - ai-provider-lanes + - ai-graphql-naming + - ai-postgres-parity + - alternate-features + - router + - router-msrv + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Require every release validation lane to have succeeded + # A failed or cancelled dependency already skips this job. The explicit + # assertion also fails the release when a lane is skipped, so a future + # condition on a validation job cannot quietly narrow what an immutable + # release was verified against. + env: + LANE_RESULTS: ${{ toJSON(needs) }} + run: | + set -euo pipefail + unsuccessful=$( + jq -r 'to_entries[] | select(.value.result != "success") | "\(.key): \(.value.result)"' \ + <<<"${LANE_RESULTS}" + ) + if [[ -n "${unsuccessful}" ]]; then + echo "release: validation lanes did not succeed:" >&2 + echo "${unsuccessful}" >&2 + exit 1 + fi + jq -r 'to_entries[] | "release: \(.key) succeeded"' <<<"${LANE_RESULTS}" + + release: + name: Publish immutable workspace release + needs: + - guard + - validate + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@v5 + with: + ref: ${{ inputs.target_ref }} + fetch-depth: 0 + - name: Restore release tag visibility + env: + TARGET_REF: ${{ inputs.target_ref }} + run: | + set -euo pipefail + # The release identity proof - current main, unused workspace tag, + # router approval - runs once in the guard job before any validation + # lane, exactly as it did before validation was split. This job only + # re-establishes the tag data that the manifest generator and the tag + # step read, and confirms this runner checked out the validated + # commit. + git fetch origin main --tags --force + test "$(git rev-parse HEAD)" = "${TARGET_REF}" + - name: Reclaim runner disk + # Only the optional router artifact compiles anything in this job. + if: inputs.include_router_artifact + uses: ./.github/actions/reclaim-runner-disk + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 - name: Generate deterministic release bundle env: RELEASE_ID: ${{ inputs.release_id }} diff --git a/docs/development/testing.md b/docs/development/testing.md index 4f8a7142..2e5f133f 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -71,8 +71,10 @@ scripts/check-ai-provider-lanes.sh doc The runner covers `provider-openai`, `provider-anthropic`, `provider-xai`, `provider-ollama`, `provider-openai-compatible`, `local-harness`, and `provider-codex-app-server`, each with only SQLite and that provider enabled. -Pass one provider feature as the second argument for a focused run. These local -commands are release evidence; a hosted workflow result is not a substitute. +Pass one provider feature as the second argument for a focused run, and +`--list` to print the lane set as JSON. The release workflow uses both to give +every provider lane its own runner and its own `target/`. These local commands +are release evidence; a hosted workflow result is not a substitute. Router changes also require warnings-denied package lanes and the dependency boundary check: diff --git a/docs/operations/release/process.md b/docs/operations/release/process.md index 99852721..b4b6a35e 100644 --- a/docs/operations/release/process.md +++ b/docs/operations/release/process.md @@ -125,12 +125,15 @@ Run **Workspace release** manually and supply: - `include_router_artifact`: normally false for source-only releases; and - `router_distribution_approval`: required when a router binary is attached. -The protected `release` environment is the human authorization boundary. The +The protected `release` environment is the human authorization boundary and +gates the workflow's entry job, so no release lane runs before approval. The workflow then: 1. proves the requested commit is current `main` and the workspace tag is new; 2. reruns documentation, dependency, package, backend, provider, Clippy, and - Rustdoc release lanes with the lockfile fixed; + Rustdoc release lanes with the lockfile fixed, as parallel jobs that each + own their `target/`, one per AI provider lane, all of which must succeed + before anything is tagged or published; 3. generates the canonical JSON manifest and Markdown release notes; 4. optionally builds the approved Linux router executable and its CycloneDX inventory; diff --git a/scripts/check-ai-provider-lanes.sh b/scripts/check-ai-provider-lanes.sh index 1c70927f..04629e9c 100755 --- a/scripts/check-ai-provider-lanes.sh +++ b/scripts/check-ai-provider-lanes.sh @@ -1,8 +1,13 @@ #!/usr/bin/env bash set -euo pipefail -if [[ $# -gt 2 ]]; then +usage() { echo "usage: scripts/check-ai-provider-lanes.sh [test|check|clippy|doc] [provider-feature]" >&2 + echo " scripts/check-ai-provider-lanes.sh --list" >&2 +} + +if [[ $# -gt 2 ]]; then + usage exit 2 fi @@ -18,6 +23,25 @@ providers=( provider-codex-app-server ) +# `--list` prints the provider lanes as a JSON array so a caller that fans the +# lanes across separate runners, each with its own `target/`, derives the set +# from this script instead of restating it. +if [[ "${mode}" == "--list" ]]; then + if [[ $# -gt 1 ]]; then + usage + exit 2 + fi + printf '[' + for index in "${!providers[@]}"; do + if [[ "${index}" -gt 0 ]]; then + printf ',' + fi + printf '"%s"' "${providers[${index}]}" + done + printf ']\n' + exit 0 +fi + case "${mode}" in test|check|clippy|doc) ;; *)