From 6ff9cca24514e21cea432da9af2fa07db12646b3 Mon Sep 17 00:00:00 2001 From: Zhe Feng Date: Tue, 13 Oct 2026 21:38:46 +0800 Subject: [PATCH 1/2] fix: resolve the openkal index and runtime revisions once per run The four build jobs of openkal-cross.yml each cloned `main` of the index and of the runtime of their own accord, at four different moments on four runners that become available at four different times, and the run jobs then asserted that all four had built the same sources. On 2026-10-13 that assertion failed for a reason the change under test had nothing to do with: run 38399400601 started at 12:43 UTC, its build jobs at 12:52:08 through 13:01:33, and mcpp-index@main took four commits inside that window (21b9a77a 12:56:12, d3a328c8 13:04:42, cfb808bc 13:10:06, 0340de61 13:17:58), so the four clones straddled pushes and the run jobs reported `index sources differ`. Resolve both refs once, in a new `revisions` job, and have every host materialise the commit it names. OPENKAL_BRANCH keeps its meaning: an override still verifies exactly what it named, now as one commit rather than as whatever the branch had become when each runner woke up. The run jobs keep their cross-host comparison; it is what proves a host used the pin rather than re-resolving for itself. git_clone_retry.sh grows the two primitives that needs: `--rev `, which prints the commit a ref names without downloading the tree, and `--commit `, which materialises exactly that commit, both under the retry the helper already exists for. Writing a SHA into the workflow instead would be the pin-that-keeps-a-check-green-by-keeping-it- out-of-date that the file itself warns about. --- .github/tools/git_clone_retry.sh | 125 ++++++++++++++++++++++++---- .github/workflows/openkal-cross.yml | 91 +++++++++++++++++++- 2 files changed, 198 insertions(+), 18 deletions(-) diff --git a/.github/tools/git_clone_retry.sh b/.github/tools/git_clone_retry.sh index a5c190605..d4d25f742 100755 --- a/.github/tools/git_clone_retry.sh +++ b/.github/tools/git_clone_retry.sh @@ -38,6 +38,8 @@ # and their attention, a slow true red costs 50 seconds. # # Usage: git_clone_retry.sh [git clone args...] +# git_clone_retry.sh --rev # print the commit it names +# git_clone_retry.sh --commit # that commit, exactly # Env: GIT_CLONE_ATTEMPTS (default 4), GIT_CLONE_BACKOFF (default "5 15 30") set -euo pipefail @@ -50,28 +52,121 @@ read -r -a BACKOFF <<< "${GIT_CLONE_BACKOFF:-5 15 30}" # different reason than the first and the log stops making sense. dest="${*: -1}" +# `cmd || rc=$?`, NOT `if cmd; then …; fi; rc=$?`. +# +# After an `if`, `$?` is the status of the IF STATEMENT — which is 0 when the +# body did not run. Written that way this script exits 0 on a clone that never +# succeeded, i.e. it turns a hard failure into a green step with a missing +# checkout. Caught by asserting the exit code in the helper's own test rather +# than by reading the output, which looked correct. +# +# ONE COPY OF IT, for all three verbs below. They back off over the same +# sequence and give up in the same words, so the retry is a property of the +# helper rather than of whichever verb was written last. +failed() { # $1 = attempt number (1-based), $2 = exit status of that attempt + if (( $1 == ATTEMPTS )); then + echo "git_clone_retry: giving up after $ATTEMPTS attempts (last exit $2)" >&2 + exit "$2" + fi + delay="${BACKOFF[$(( $1 - 1 < ${#BACKOFF[@]} ? $1 - 1 : ${#BACKOFF[@]} - 1 ))]}" + echo "git_clone_retry: attempt $1/$ATTEMPTS failed (exit $2); retrying in ${delay}s" >&2 + sleep "$delay" +} + +# ── TWO MORE VERBS, AND WHY THEY ARE IN THIS FILE ────────────────────────── +# +# `--rev` and `--commit` exist because "clone this branch" stopped being one +# act the moment four jobs needed it. Every `build` job of `openkal-cross.yml` +# cloned `main` of the index and of the runtime of its own accord, on runners +# that become available at different moments — the macOS leg waits in a queue +# the others do not — and the `run` job then asserted that all four had built +# the same sources, and failed the run when they had not. On 2026-10-13 that +# assertion was right to fail and the change under test had nothing to do with +# it: the run started at 12:43 UTC, its build jobs at 12:52:08 through +# 13:01:33, and mcpp-index@main took four commits between 12:56:12 and +# 13:17:58 — so clones taken minutes apart straddled pushes, and the run job +# reported `index sources differ`. +# +# The repair is to resolve the ref ONCE, at the start of the run, and have +# every host materialise that commit. Writing the SHA into the workflow would +# be the pin nobody is reminded to move that `openkal-cross.yml` warns about in +# its own header, so the resolution is a job — and these are the two primitives +# it needs: ask what a ref names, and get exactly that commit. + +# `--rev `: print the commit the ref names. +# +# `ls-remote` rather than clone-and-rev-parse: what the caller wants is one +# number, and the tree behind it is tens of megabytes on both of the +# repositories this is pointed at. +if [[ "${1:-}" == "--rev" ]]; then + [ "$#" -eq 3 ] || { echo "git_clone_retry: usage: --rev " >&2; exit 2; } + ref="$2"; url="$3" + # A bare name is a BRANCH. `refs/…` and `HEAD` are taken as written, so a + # tag or a fully qualified ref can be asked for too. + case "$ref" in + refs/*|HEAD) want="$ref" ;; + *) want="refs/heads/$ref" ;; + esac + for (( i = 1; i <= ATTEMPTS; i++ )); do + rc=0 + # stderr is NOT captured: what git says about a URL it cannot reach is + # the diagnostic, and it is the message a plain clone would have shown. + listing=$(git ls-remote "$url" "$want") || rc=$? + if (( rc == 0 )); then + sha=$(printf '%s\n' "$listing" | awk -v w="$want" '$2 == w { print $1; exit }') + # No such ref is an ANSWER, not a blip: the host was reached and + # said the ref is not there, so retrying only delays the report. + [ -n "$sha" ] || { echo "git_clone_retry: $url has no $want" >&2; exit 1; } + printf '%s\n' "$sha" + exit 0 + fi + failed "$i" "$rc" + done +fi + +# `--commit `: materialise exactly that commit, and nothing +# the branch did afterwards. +# +# NOT `git clone --branch `: that argument takes a branch or a tag, and a +# commit is neither. `init` + `fetch --depth 1 origin ` is the shape that +# names a commit, and however far the branch moves after the resolution, it +# costs one commit's tree once. +if [[ "${1:-}" == "--commit" ]]; then + [ "$#" -eq 4 ] || { echo "git_clone_retry: usage: --commit " >&2; exit 2; } + sha="$2"; url="$3" dest="$4" + [[ "$sha" =~ ^[0-9a-f]{40}$ ]] \ + || { echo "git_clone_retry: --commit wants a 40-hex commit, got '$sha'" >&2; exit 2; } + for (( i = 1; i <= ATTEMPTS; i++ )); do + rc=0 + { + git -c init.defaultBranch=main init --quiet "$dest" && + git -C "$dest" remote add origin "$url" && + git -C "$dest" fetch --quiet --depth 1 origin "$sha" && + git -C "$dest" checkout --quiet --detach FETCH_HEAD + } || rc=$? + if (( rc == 0 )); then + exit 0 + fi + # Nothing a failed attempt made survives it, for the reason the clone + # path gives below about its partial clone: a tree half-made by this + # attempt is not a retry, it is a second failure with a different cause. + # And a caller that gives up should not find a directory claiming to be + # the checkout it asked for. + rm -rf -- "$dest" + failed "$i" "$rc" + done +fi + for (( i = 1; i <= ATTEMPTS; i++ )); do - # `cmd || rc=$?`, NOT `if cmd; then …; fi; rc=$?`. - # - # After an `if`, `$?` is the status of the IF STATEMENT — which is 0 when - # the body did not run. Written that way this script exits 0 on a clone that - # never succeeded, i.e. it turns a hard failure into a green step with a - # missing checkout. Caught by asserting the exit code in the helper's own - # test rather than by reading the output, which looked correct. rc=0 git clone "$@" || rc=$? if (( rc == 0 )); then exit 0 fi - if (( i == ATTEMPTS )); then - echo "git_clone_retry: giving up after $ATTEMPTS attempts (last exit $rc)" >&2 - exit "$rc" - fi - if [[ -n "$dest" && "$dest" != -* && -e "$dest" ]]; then + # Nothing to remove on the last attempt: there is no retry to prepare for. + if (( i < ATTEMPTS )) && [[ -n "$dest" && "$dest" != -* && -e "$dest" ]]; then echo "git_clone_retry: removing partial '$dest' before retrying" >&2 rm -rf -- "$dest" fi - delay="${BACKOFF[$(( i - 1 < ${#BACKOFF[@]} ? i - 1 : ${#BACKOFF[@]} - 1 ))]}" - echo "git_clone_retry: attempt $i/$ATTEMPTS failed (exit $rc); retrying in ${delay}s" >&2 - sleep "$delay" + failed "$i" "$rc" done diff --git a/.github/workflows/openkal-cross.yml b/.github/workflows/openkal-cross.yml index f79b40c91..827fb43c8 100644 --- a/.github/workflows/openkal-cross.yml +++ b/.github/workflows/openkal-cross.yml @@ -44,6 +44,13 @@ name: openkal cross-build (4 hosts × 3 targets) # Three cells are native. The nine crosses include every output built on Linux # ARM64, and are what a single-host workflow cannot reach. # +# A `revisions` JOB RUNS FIRST, before any of them, and resolves the index and +# the runtime branches to one commit each. The four build jobs materialise +# those commits, because "the four hosts built the same sources" — the claim +# the run jobs assert — needs one revision to be about. Four independent +# clones of a moving `main` are not that, and on 2026-10-13 they were not one +# revision (run 38399400601). See the job. +# # THE RUN JOBS INSTALL NOTHING — not mcpp, not a compiler, not a C runtime. # A program above openkal carries its C library, its C++ runtime and its # unwinder; what remains is the operating system it was built for. If a @@ -93,8 +100,75 @@ env: OPENKAL_BRANCH: ${{ github.event.inputs.openkal_ref || 'main' }} jobs: + # ── THE TWO REVISIONS EVERY HOST BUILDS AGAINST, RESOLVED ONCE ─────────── + # + # FOUR JOBS CLONING ONE BRANCH ARE NOT FOUR COPIES OF ONE REVISION. The build + # matrix cloned `main` of the index and of the runtime of its own accord, on + # runners that become available at different moments — the macOS leg waits in + # a queue the others do not — and the `run` job below then asserted that all + # four had built the same sources. + # + # THAT ASSERTION WAS RIGHT AND IT WENT RED FOR A REASON NO HOST CAUSED. On + # 2026-10-13, run 38399400601, all three `run` jobs failed with + # + # index sources differ + # + # — so the four hosts did record different revisions. The run started at + # 12:43 UTC and its build jobs at 12:52:08, 12:53:44, 12:58:20 and 13:01:33 + # UTC, and they do not clone at the same point in their own step sequence — + # the macOS leg builds this commit's mcpp first — while mcpp-index@main took + # four commits inside that window: 21b9a77a at 12:56:12, d3a328c8 at 13:04:42, + # cfb808bc at 13:10:06, 0340de61 at 13:17:58. Two clones straddling one push + # is all it takes, and the reading arrives 40 minutes later, in a job that + # built nothing. + # + # THE ALTERNATIVE — pinning a SHA into this file — is the failure + # `OPENKAL_BRANCH` above is annotated with: a pin nobody is reminded to move, + # which keeps a check green by keeping it out of date. So the refs are + # resolved here, once, and every host materialises the commit named below. + # `OPENKAL_BRANCH` keeps its meaning: a run that overrides it still verifies + # exactly what it named, now as one commit rather than as whatever the branch + # had become when each runner woke up. + # + # The `run` job keeps its comparison. It is no longer a race against a moving + # branch; it is what proves each host used the pin rather than re-resolving + # for itself. + revisions: + name: the index and runtime revisions this run builds + runs-on: ubuntu-24.04 + timeout-minutes: 10 + outputs: + index: ${{ steps.resolve.outputs.index }} + runtime: ${{ steps.resolve.outputs.runtime }} + defaults: + run: + shell: bash + steps: + - uses: actions/checkout@v4 + + - name: Resolve both refs, once + id: resolve + run: | + set -euo pipefail + RETRY="$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" + index=$("$RETRY" --rev main https://github.com/mcpp-community/mcpp-index) + runtime=$("$RETRY" --rev "$OPENKAL_BRANCH" \ + https://github.com/mcpplibs/openkal-llvm-runtime) + # The `run` job's comparison already refuses a SHA that is not 40 hex; + # this refuses to PIN one, so a mis-resolution fails here, in 10 + # seconds, rather than as four builds of four different trees. + for commit in "$index" "$runtime"; do + [[ "$commit" =~ ^[0-9a-f]{40}$ ]] \ + || { echo "::error::not a commit: '$commit'"; exit 1; } + done + echo "mcpp-index@main is $index" + echo "openkal-llvm-runtime@$OPENKAL_BRANCH is $runtime" + echo "index=$index" >> "$GITHUB_OUTPUT" + echo "runtime=$runtime" >> "$GITHUB_OUTPUT" + build: name: build 3 targets on ${{ matrix.host }} + needs: revisions runs-on: ${{ matrix.runner }} timeout-minutes: 120 strategy: @@ -238,6 +312,9 @@ jobs: command -v x86_64-w64-mingw32-gcc gcc 2>/dev/null || true - name: The program — one source, three targets + env: + INDEX_SHA: ${{ needs.revisions.outputs.index }} + RUNTIME_SHA: ${{ needs.revisions.outputs.runtime }} run: | set -euo pipefail # THE SEVENTH CALL SITE. `git_clone_retry.sh` was written because a @@ -251,11 +328,16 @@ jobs: # # One name that did not resolve ended a 120-minute job in its first # minute, beside a real failure it had nothing to do with. + # + # BOTH CLONES NOW NAME A COMMIT. `--depth 1 -b main` was four jobs + # describing a moving branch as one revision; see the `revisions` job + # for the run that paid for it. "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ - --quiet --depth 1 -b "$OPENKAL_BRANCH" \ + --commit "$RUNTIME_SHA" \ https://github.com/mcpplibs/openkal-llvm-runtime "$RUNNER_TEMP/okl" "$GITHUB_WORKSPACE/.github/tools/git_clone_retry.sh" \ - --quiet --depth 1 https://github.com/mcpp-community/mcpp-index "$RUNNER_TEMP/openkal-index" + --commit "$INDEX_SHA" \ + https://github.com/mcpp-community/mcpp-index "$RUNNER_TEMP/openkal-index" export MCPP_OPENKAL_INDEX="$RUNNER_TEMP/openkal-index" cd "$RUNNER_TEMP/okl/examples/same-source" mkdir -p "$RUNNER_TEMP/out" @@ -728,6 +810,9 @@ jobs: ecosystem-native-arm64: name: openkal native stack (Linux ARM64, LLVM 23.1.3) + # The same index revision the build jobs pinned, not whatever `main` had + # become when this runner woke up; see the `revisions` job. + needs: revisions runs-on: ubuntu-24.04-arm timeout-minutes: 45 defaults: @@ -743,7 +828,7 @@ jobs: - uses: actions/checkout@v4 with: repository: mcpp-community/mcpp-index - ref: main + ref: ${{ needs.revisions.outputs.index }} path: _openkal-native-index - name: Native platform and graph stack From c28b53ac359befe10770479c5da5b6830b4fdde8 Mon Sep 17 00:00:00 2001 From: Zhe Feng Date: Thu, 15 Oct 2026 15:50:58 +0800 Subject: [PATCH 2/2] fix: the wine install survives dpkg's shared-doc-file refusal The job died on 2026-10-15 (run 38695300904) in apt's unpack of wine's i386 closure, before a single byte was compiled: dpkg: error processing archive .../libxml2_2.9.14+dfsg-1.3ubuntu3.10_i386.deb (--unpack): trying to overwrite shared '/usr/share/doc/libxml2/changelog.Debian.gz', which is different from other instances of package libxml2:i386 libpng16-16t64, libx11-6 and libfreetype6, all alike. Multi-Arch: same instances must share every file bit-for-bit, and the i386 and amd64 debs in the Ubuntu archive embed changelogs that differ (Launchpad #1574351 and its kin). That is a property of the archive, not of the runner: the step's own `|| sudo apt-get install -y wine` retry failed with the identical error, so a rerun re-rolls nothing. Give dpkg --force-overwrite on every install this step makes, the cold path and the cached-debs path alike. The changelogs are not what this job reads; wine is. --- .github/workflows/cross-build-test.yml | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index ff412499e..936293701 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -270,10 +270,13 @@ jobs: - name: Install wine timeout-minutes: 20 run: | + # PASSED TO DPKG ON EVERY UNPACK BELOW; see the cold path's note for + # the run that paid for it. + DPKG_OVERWRITE='-o Dpkg::Options::=--force-overwrite' sudo dpkg --add-architecture i386 || true if ls ~/wine-debs/*.deb >/dev/null 2>&1; then sudo dpkg -i ~/wine-debs/*.deb 2>/dev/null \ - || { sudo apt-get update -qq; sudo apt-get install -f -y; } + || { sudo apt-get update -qq; sudo apt-get install -f -y $DPKG_OVERWRITE; } else # The runner image carries third-party apt lists (Google Chrome # among them) that this job does not use, and a transient @@ -293,7 +296,25 @@ jobs: || sudo apt-get install -y --download-only wine mkdir -p ~/wine-debs cp /var/cache/apt/archives/*.deb ~/wine-debs/ 2>/dev/null || true - sudo apt-get install -y wine64 wine || sudo apt-get install -y wine + # OVERWRITE THE SHARED DOC FILES. The install died here on + # 2026-10-15 (run 38695300904) in dpkg's unpack, still before a + # single byte was compiled: + # + # dpkg: error processing archive + # .../libxml2_2.9.14+dfsg-1.3ubuntu3.10_i386.deb (--unpack): + # trying to overwrite shared + # '/usr/share/doc/libxml2/changelog.Debian.gz', which is + # different from other instances of package libxml2:i386 + # + # libpng16-16t64, libx11-6 and libfreetype6, all alike. Multi-Arch: + # same instances must share every file bit-for-bit, and the i386 and + # amd64 debs in Ubuntu's archive embed changelogs that differ + # (Launchpad #1574351 and its kin). That is a property of the + # archive, not of this runner: the step's own retry below it failed + # with the identical error, so a rerun re-rolls nothing. The + # changelogs are not what this job reads; wine is. + sudo apt-get install -y $DPKG_OVERWRITE wine64 wine \ + || sudo apt-get install -y $DPKG_OVERWRITE wine fi wine --version