From 02bf9c4fa30ff87e15bd5f246bbb0e5b844497a1 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Mon, 24 Aug 2026 10:56:37 +0200 Subject: [PATCH] Bound the qemu install in the nightly too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lane installs qemu with sudo apt-get update && sudo apt-get install -y qemu-user-static which is the exact line that sat for the better part of an hour in go-crdt/crdt, on the same runner image, more than once. Bounding it there identified the culprit: three attempts of ten minutes each produced not one line of output from apt-get update, and the install was never reached. The runner image already ships apt's lists, so the refresh was not needed at all. It has not hung here. But this lane runs once a night, takes forty minutes when it works, and has just come back from seventeen consecutive red nights — an hour lost to apt would read as another one, and nobody would know which kind of red it was until they opened the log. Same shape as the timeout in #436: what decides whether a slow-but- correct run counts should not be something unrelated to the code. Co-Authored-By: Claude Opus 5 --- .github/workflows/arch-qemu-nightly.yml | 31 ++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/.github/workflows/arch-qemu-nightly.yml b/.github/workflows/arch-qemu-nightly.yml index df5c22ff..202cf67e 100644 --- a/.github/workflows/arch-qemu-nightly.yml +++ b/.github/workflows/arch-qemu-nightly.yml @@ -44,7 +44,36 @@ jobs: - uses: actions/setup-go@v6 with: go-version: stable - - run: sudo apt-get update && sudo apt-get install -y qemu-user-static + # This has not hung here. It hung in go-crdt/crdt, on the same runner + # image and the same one-line install, for the better part of an hour and + # more than once — and bounding it is what identified the culprit: three + # attempts of ten minutes each produced not one line of output from + # apt-get update, while the install itself was never reached. The cure is + # copied rather than waited for, because a lane that runs once a night + # and takes forty minutes has no spare hour to lose to apt. + # + # The runner image already ships apt's package lists, so the install + # usually needs no refresh at all. It is tried on the lists that are there + # first, and they are only refreshed if that was not enough. apt is also + # told to give up on a stuck mirror rather than sit on it, which is what + # made the outer timeout the only thing stopping it. + - name: Install qemu-user-static + run: | + apt=(-y -qq -o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30) + if timeout 240 sudo apt-get install "${apt[@]}" qemu-user-static; then + exit 0 + fi + for attempt in 1 2; do + echo "::warning::the runner's package lists were not enough; refreshing them (attempt $attempt)" + timeout 240 sudo apt-get update "${apt[@]}" || true + sudo rm -f /var/lib/apt/lists/lock /var/cache/apt/archives/lock /var/lib/dpkg/lock* + sudo dpkg --configure -a || true + if timeout 240 sudo apt-get install "${apt[@]}" qemu-user-static; then + exit 0 + fi + done + echo "::error::qemu-user-static could not be installed" + exit 1 - name: Test under qemu (${{ matrix.goarch }}) # qemu-user emulation is ~50-100x slower than native, so the wall here is # not a hang detector: it is the only thing that decides whether a