Repository navigation
Give the emulated lanes a wall they are not already touching - #436
Merged
Merged
Conversation
The nightly on main died with
panic: test timed out after 30m0s
running tests:
TestStringToIBase (0s)
and no failing test: the s390x lane hit the 30-minute wall at 1801.594s.
The wall here is not a hang detector. qemu-user is 50-100x slower than
native, so it is the only thing deciding whether a slow-but-correct run
counts, and it has to sit well clear of what the run actually takes.
Measured on internal/vm, the heaviest package, in run 32661057000 — the
verification run, all four green:
loong64 990s riscv64 1204s
ppc64le 1284s s390x 1754s <- 46s under an 1800s wall
A 2.6% margin, which ordinary runner variance crosses, and the very next
run of that same tree crossed it. So the red was the wall rather than the
code: nothing about the three fixes that landed today is implicated, and
the three arches that were never near the wall stayed green.
60m puts s390x at about half of it, which is what the comment above the
line already claimed. The numbers are in the comment now, so the next
person to look does not have to measure them again to know how much room
there is.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tannevaled
added a commit
that referenced
this pull request
Aug 24, 2026
The lane installs qemu with
sudo apt-get update && sudo apt-get install -y qemu-user-static
which is the exact line that sat for the better part of an hour in
go-crdt/crdt, on the same runner image, more than once. Bounding it there
identified the culprit: three attempts of ten minutes each produced not
one line of output from apt-get update, and the install was never
reached. The runner image already ships apt's lists, so the refresh was
not needed at all.
It has not hung here. But this lane runs once a night, takes forty
minutes when it works, and has just come back from seventeen consecutive
red nights — an hour lost to apt would read as another one, and nobody
would know which kind of red it was until they opened the log.
Same shape as the timeout in #436: what decides whether a slow-but-
correct run counts should not be something unrelated to the code.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly I dispatched on
mainafter merging #431/#432/#433 came back red on s390x. It is not a failing test:Zero
--- FAILlines. The package hit the wall:FAIL …/internal/vm 1801.594sagainst-timeout=30m.The wall was the bug
-timeouton these lanes is not a hang detector. qemu-user is 50–100× slower than native, so it is the only thing deciding whether a slow-but-correct run counts, and it has to sit well clear of what the run actually takes. Measured oninternal/vm, the heaviest package, in run 32661057000 — the verification run, all four green:A 2.6% margin, which ordinary runner variance crosses — and the very next run of that same tree crossed it, at 1801.594s.
So the red is the wall and not the code. Nothing about the three fixes that landed today is implicated, and the three arches that were never near the wall stayed green in both runs.
60m puts s390x at about half the wall, which is what the comment above the line already claimed before the number was overtaken. The measurements go into that comment, so the next person does not have to take them again to know how much room there is.
Not done here, deliberately: sharding
internal/vmacross parallel jobs would cut the wall clock, and it is complexity this nightly safety net does not need to carry for a run that finishes comfortably inside an hour.