From 8f162ceb9f0ee539afaae1c94084e686d959f4aa Mon Sep 17 00:00:00 2001 From: Matt Toohey Date: Thu, 29 Oct 2026 16:28:14 +1100 Subject: [PATCH] ci: cache the Tauri apt .debs and fit a cold Windows Rust run in its cap Fixes the two CI workflow defects that turned main red most often in the 21-28 Oct 2026 audit. No product code changes; one CI contract test is adapted to the moved package list (item 3). Pre-check: origin/main is eda26b942. #8613 changed both files since the audit, but only the Unit Tests rust-cache key and a Flake check selection step, which this change does not touch; it applies cleanly. #8613's flake-check.test.mjs does parse the Tauri package list, so it is updated here (item 3). No open PR changes how the Tauri packages are fetched, the Windows timeout, or its Check (workspace) step: - #8422, #8321 and #8373 only add packages to the list, and would now add them to TAURI_APT_PACKAGES. - #7772 bumps the setup-mold pin; #8213, #8187 and #7848 touch other _ci-rust.yml jobs. - #8414 adds two mesh-llm Tauri steps to the Windows job. If it lands, they must fit the cold budget below. 1. Desktop Tauri: cache the apt .debs (_ci-desktop.yml, desktop-tauri) Runs: 62 jobs in 51 run attempts, 2026-10-22..27. On main: 39821769576, 39822883395, 39827234118, 40333705511, 40342468627, 40348069929, 40353619362, 40368315739, 40373404267, 40374519539, 40556742929. Failure: in "Install Tauri dependencies (Linux)", all 3 ci-apt-retry.sh attempts hit their 420 s limit while azure.archive.ubuntu.com served the 62 MB download at 20-65 KB/s. The 25.5 MB libwebkit2gtk-4.1-0 .deb alone takes a whole attempt, and each attempt loses its partial download. Change: - The package list moves to a job-level TAURI_APT_PACKAGES. - New "Resolve Tauri apt archives" step: runs the existing apt-get update, then `apt-get install --print-uris -o Acquire::ForceHash=SHA256` and keeps each file's name, size and SHA256, sorted, without the mirror URI. The key is tauri-debs-- plus the first 16 hex digits of that list's sha256. - actions/cache/restore restores ~/.cache/tauri-debs by the exact key, falling back to the prefix (same v5 SHA as the pnpm and Playwright caches). - "Install Tauri dependencies (Linux)" keeps its name. It copies the restored files whose SHA256 matches the list into /var/cache/apt/archives, runs the ci-apt-retry.sh install unchanged, then rebuilds ~/.cache/tauri-debs with exactly the listed files. The rebuild is best-effort: after the packages are installed, a failure only warns (::warning::) and does not set rebuilt=true. - actions/cache/save runs only on core-lane pushes without an exact hit, and only when the rebuild succeeded, so a partial directory is never saved under the exact key. PR runs never save. Why the hash check: apt 2.8.3 (noble) does not check a cached .deb against the signed index. It reuses an archived .deb of the expected size without hashing it. A corrupted .deb of the same size in a restored cache was used as is ("Need to get 0 B/334 kB"), and only zstd's own checksum stopped dpkg. So the install step checks each restored file with sha256sum against the SHA256 that --print-uris reads from the signed index. --print-uris prints MD5Sum unless ForceHash is set. Verification: Ubuntu 24.04 amd64 containers on OrbStack, apt 2.8.3, running the Resolve and Install scripts verbatim (extracted from this workflow with Ruby's YAML) as a non-root sudo user, one fresh container per simulated job, with the cache restore emulated (exact key, then the newest prefix match) and the save step's if: applied. The bare image lacks the runner's preinstalled packages, so it resolves 436 files (254 MB), not 62 MB. - core, pull_request, empty cache: miss, "Need to get 254 MB". The install passes and nothing is saved. - core, push, empty cache: same key (tauri-debs-ubuntu24.04-89a18607a37aa494), saved. - mesh, push, with eth0 down for the install step: exact hit, "Need to get 0 B/254 MB", 0 mirror fetches, passes. The save is skipped. - compiled-flags, pull_request: exact hit, "Need to get 0 B/254 MB". - The resolved lists from all four jobs are byte-identical. - restore-keys fallback with one listed .deb missing, one corrupted at the same size, and one stale extra file: sha256sum rejects the corrupted file, apt fetches only that one and the missing one ("Need to get 356 kB/254 MB"), the stale file is dropped, and the saved entry is byte-identical to the original. - Rebuild step alone (GNU cp, macOS bash 3.2): all files present -> rebuilt=true, stale file dropped; one missing -> warning, step exits 0, no output. Only hosted CI can show the runner's real list (about 62 MB), the same key in all three lanes, and the actions/cache round trip. The first PR run misses because main has no key yet; after merge the first main push saves it. Remaining risk: after Ubuntu updates a large package, runs fetch it from the mirror until a main push saves a new key. 2. Windows Rust: drop the duplicate check, cap at 60 min (_ci-rust.yml) Runs, all on main: 39796569342 (job 119885795086), 39821769576 (119975358772), 39827234118 (119994286695), 39865872891 (120128351939) and 40402556064 (121852634685, right after the Rust 1.98.1 bump). Every test passed. The 45 min cap then cancelled "Post Run Swatinem/rust-cache" 1.6-3.7 min into its save, so no cache was written and the next push was cold again. Measured from the jobs API: without a cache the steps took 41.3-43.5 min (clippy 17.2-18.0, Check (workspace) 2.0-2.8, buzz-dev-mcp 2.0-2.4, ACP 1.2-1.3, buzz-agent 0.8-1.1, Tauri check 6.1-6.5, Tauri tests 10.2-11.4). The only cold save that completed, in 40011714848 (job 120614064303), took 4.4 min. Change: - Delete Check (workspace). The clippy step before it covers the same --workspace --all-targets --target set and type-checks everything cargo check does, with -D warnings on top. This saves 2.0-2.8 min. - Raise timeout-minutes from 45 to 60, with the measured breakdown in the comment: 39-41 min of steps plus a ~4.5 min save, so ~15 min of margin. Warm main runs (15.1-22.5 min) are unaffected. Only hosted CI can show a cold run saving its cache inside the new cap; the next toolchain bump or cache eviction will be the test. 3. Flake check test: read the core lane's packages from TAURI_APT_PACKAGES (scripts/flake-check.test.mjs) #8613 added "repeat invocations mirror the lanes they stand in for", which checks that the advisory flake-tauri lane installs the same apt packages as the Desktop Tauri core lane by matching the multi-line `apt-get install -y --no-install-recommends \` list in _ci-desktop.yml. That list now lives in TAURI_APT_PACKAGES, so the regex matched nothing and the test failed ("Desktop Tauri apt packages"). It now reads TAURI_APT_PACKAGES and compares package names with the flake lane's list. Renaming one package in _ci-flake.yml still fails it. The flake-tauri lane keeps its own uncached install; giving it the same read-only .deb restore is a follow-up. Checks: - actionlint 1.7.12 with shellcheck 0.11.0 on _ci-desktop.yml and _ci-rust.yml: clean. - node --test scripts/ci-selection.test.mjs scripts/ci-required-gates.test.mjs scripts/ci-apt-retry.test.mjs scripts/flake-check.test.mjs (macOS, Node 24.15.0, PATHS_FILTER_ACTION at the pinned dorny/paths-filter dist): 84 pass, 0 fail, 8 skipped (the Linux-only apt-retry tests). Before item 3, flake-check.test.mjs failed 1 test on this branch. - scripts/test-rust-cache-contract.sh, its -regressions.sh and scripts/test-ci-required-context-isolation.sh pass. Co-Authored-By: Claude Opus 5.5 Co-Authored-By: Claude Fable 5.1 Signed-off-by: Matt Toohey --- .github/workflows/_ci-desktop.yml | 92 ++++++++++++++++++++++++++----- .github/workflows/_ci-rust.yml | 13 +++-- scripts/flake-check.test.mjs | 20 ++++--- 3 files changed, 98 insertions(+), 27 deletions(-) diff --git a/.github/workflows/_ci-desktop.yml b/.github/workflows/_ci-desktop.yml index 90eacc59f1e..bcdb3b63d96 100644 --- a/.github/workflows/_ci-desktop.yml +++ b/.github/workflows/_ci-desktop.yml @@ -69,7 +69,7 @@ jobs: desktop-tauri: name: Desktop Tauri (${{ matrix.lane }}) runs-on: ubuntu-latest - # A lane needs up to ~15m. ci-apt-retry.sh bounds the apt step at ~34m worst + # A lane needs up to ~15m. ci-apt-retry.sh bounds the apt steps at ~34m worst # case (update ~8.5m + install ~25m); 55m fits both, so a degraded mirror # fails with the wrapper's error instead of the job timeout. timeout-minutes: 55 @@ -93,6 +93,22 @@ jobs: run: just desktop-tauri-test-compiled-flags permissions: contents: read + env: + # Resolved and installed by the steps below; the .deb cache key hashes + # what apt resolves for this list. + TAURI_APT_PACKAGES: >- + build-essential + curl + file + libasound2-dev + libayatana-appindicator3-dev + libgtk-3-dev + librsvg2-dev + libssl-dev + libwebkit2gtk-4.1-dev + libxdo-dev + patchelf + wget steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 with: @@ -104,24 +120,70 @@ jobs: workspaces: desktop/src-tauri shared-key: ${{ matrix.cache }} save-if: ${{ matrix.save-cache && github.event_name != 'pull_request' }} + # A degraded Ubuntu mirror serves .debs at ~50 KB/s, and the ~62 MB + # Tauri download (one 25.5 MB libwebkit2gtk .deb) then outlasts every + # ci-apt-retry.sh attempt. Cache the .debs in the Actions cache so the + # mirror only serves `apt-get update` and packages Ubuntu has updated + # since main last saved. + - name: Resolve Tauri apt archives + id: tauri-debs + run: | + set -euo pipefail + CI_APT_ATTEMPT_SECONDS=90 scripts/ci-apt-retry.sh sudo apt-get update + read -ra packages <<< "$TAURI_APT_PACKAGES" + # Name, size and SHA256 (from the signed index) of each .deb the + # install will fetch. The mirror URI is left out so the key doesn't + # depend on which mirror answered. Runs before the restore: + # --print-uris omits files that are already in + # /var/cache/apt/archives. + sudo apt-get install --print-uris -qq -y --no-install-recommends \ + -o Acquire::ForceHash=SHA256 "${packages[@]}" | + awk '{print $2, $3, $4}' | sort > "$RUNNER_TEMP/tauri-debs.txt" + . /etc/os-release + prefix="tauri-debs-${ID}${VERSION_ID}-" + echo "prefix=$prefix" >> "$GITHUB_OUTPUT" + echo "key=$prefix$(sha256sum "$RUNNER_TEMP/tauri-debs.txt" | cut -c1-16)" >> "$GITHUB_OUTPUT" + - name: Restore Tauri apt archives + id: tauri-debs-cache + uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5 + with: + path: ~/.cache/tauri-debs + key: ${{ steps.tauri-debs.outputs.key }} + restore-keys: ${{ steps.tauri-debs.outputs.prefix }} - name: Install Tauri dependencies (Linux) + id: tauri-debs-install env: DEBIAN_FRONTEND: noninteractive run: | - CI_APT_ATTEMPT_SECONDS=90 scripts/ci-apt-retry.sh sudo apt-get update - scripts/ci-apt-retry.sh sudo apt-get install -y --no-install-recommends \ - build-essential \ - curl \ - file \ - libasound2-dev \ - libayatana-appindicator3-dev \ - libgtk-3-dev \ - librsvg2-dev \ - libssl-dev \ - libwebkit2gtk-4.1-dev \ - libxdo-dev \ - patchelf \ - wget + set -euo pipefail + debs=~/.cache/tauri-debs + mkdir -p "$debs" + # apt reuses an archived .deb of the expected size without checking + # its hash, so seed only restored files whose SHA256 matches the + # signed index. apt downloads and verifies the rest. sha256sum exits + # non-zero when a file fails or none is restored; neither is fatal. + awk '{sub(/^SHA256:/, "", $3); print $3 " " $1}' "$RUNNER_TEMP/tauri-debs.txt" > "$RUNNER_TEMP/tauri-debs.sha256" + (cd "$debs" && { sha256sum --check --ignore-missing "$RUNNER_TEMP/tauri-debs.sha256" || true; }) | + sed -n 's/: OK$//p' | (cd "$debs" && xargs -r sudo cp -t /var/cache/apt/archives/) + read -ra packages <<< "$TAURI_APT_PACKAGES" + scripts/ci-apt-retry.sh sudo apt-get install -y --no-install-recommends "${packages[@]}" + # Keep exactly the resolved files, so a restore-keys fallback can't + # pile up superseded versions. The packages are installed by now, so + # a failure here only warns and skips the save. + if rm -rf "$debs" && mkdir -p "$debs" && + awk '{print "/var/cache/apt/archives/" $1}' "$RUNNER_TEMP/tauri-debs.txt" | xargs -r cp -t "$debs"; then + echo "rebuilt=true" >> "$GITHUB_OUTPUT" + else + echo "::warning::Could not rebuild the Tauri apt archive cache; not saving it." + fi + # Like the other caches here, only pushes save; PR runs restore. All + # three lanes resolve the same key, so one lane saves it. + - name: Save Tauri apt archives + if: matrix.lane == 'core' && github.event_name == 'push' && steps.tauri-debs-cache.outputs.cache-hit != 'true' && steps.tauri-debs-install.outputs.rebuilt == 'true' + uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5 + with: + path: ~/.cache/tauri-debs + key: ${{ steps.tauri-debs.outputs.key }} - name: Desktop Tauri ${{ matrix.lane }} run: ${{ matrix.run }} env: diff --git a/.github/workflows/_ci-rust.yml b/.github/workflows/_ci-rust.yml index 3190d877ae7..e536e131620 100644 --- a/.github/workflows/_ci-rust.yml +++ b/.github/workflows/_ci-rust.yml @@ -111,9 +111,12 @@ jobs: windows-rust: name: Windows Rust (x86_64-pc-windows-msvc) runs-on: windows-latest - # Windows runners are slow and this compiles the workspace + Tauri crate - # cold across four steps; budget generously. - timeout-minutes: 45 + # With no rust-cache (after a toolchain bump or eviction), the steps take + # 39-41m on main (2026-10-22..26): clippy 17-18m, Tauri check ~6m, Tauri + # tests 10-11m, the rest ~5m. The cache save that follows takes ~4.5m and + # runs inside this cap. If the cap kills it, nothing is saved and the next + # push is cold again. 60m leaves ~15m over that; warm runs take 15-23m. + timeout-minutes: 60 if: inputs.lane == 'required' && (github.event_name == 'push' || inputs.rust || inputs.desktop_rust) permissions: contents: read @@ -144,10 +147,10 @@ jobs: for bin in buzz-acp buzz-agent buzz-dev-mcp git-credential-nostr buzz; do touch "desktop/src-tauri/binaries/${bin}-${TARGET}.exe" done + # Clippy type-checks everything `cargo check` would, so no separate + # workspace check step. - name: Clippy (workspace) run: cargo clippy --workspace --all-targets --target $env:TARGET -- -D warnings - - name: Check (workspace) - run: cargo check --workspace --all-targets --target $env:TARGET - name: Test (buzz-dev-mcp) # The Windows-only bash resolver lives in buzz-dev-mcp; its unit tests # only gate if this crate is tested ON Windows. diff --git a/scripts/flake-check.test.mjs b/scripts/flake-check.test.mjs index 5450e484a17..e390c61d61c 100644 --- a/scripts/flake-check.test.mjs +++ b/scripts/flake-check.test.mjs @@ -1077,14 +1077,20 @@ test("repeat invocations mirror the lanes they stand in for", () => { runner, ); // The Tauri lane builds what the Desktop Tauri core lane does: the same - // workspace cache, system packages and sidecar stubs. + // workspace cache, system packages and sidecar stubs. The core lane lists + // its packages in TAURI_APT_PACKAGES, which its .deb cache key hashes. const desktop = read("../.github/workflows/_ci-desktop.yml"); - const packages = (text) => - /apt-get install -y --no-install-recommends \\\n((?: {12}\S+(?: \\)?\n)+)/.exec( - text, - )?.[1]; - assert.ok(packages(desktop), "Desktop Tauri apt packages"); - assert.equal(packages(flake), packages(desktop)); + const desktopPackages = /^ {6}TAURI_APT_PACKAGES: >-\n((?: {8}\S+\n)+)/m + .exec(desktop)?.[1] + .split(/\s+/) + .filter(Boolean); + const flakePackages = + /apt-get install -y --no-install-recommends \\\n((?: {12}\S+(?: \\)?\n)+)/ + .exec(flake)?.[1] + .split(/[\s\\]+/) + .filter(Boolean); + assert.ok(desktopPackages?.length, "Desktop Tauri apt packages"); + assert.deepEqual(flakePackages, desktopPackages); assert.match( flake, /flake-tauri:\n(?: {4}.*\n)*? {6}- uses: Swatinem\/rust-cache@\w+\n {8}with:\n {10}workspaces: desktop\/src-tauri\n {10}shared-key: desktop-core\n {10}save-if: false\n/,