Skip to content

Give the emulated lanes a wall they are not already touching - #436

Merged
tannevaled merged 1 commit into
mainfrom
ci/arch-timeout-headroom
Aug 24, 2026
Merged

tannevaled merged 1 commit into
mainfrom
ci/arch-timeout-headroom

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

The nightly I dispatched on main after merging #431/#432/#433 came back red on s390x. It is not a failing test:

panic: test timed out after 30m0s
        running tests:
                TestStringToIBase (0s)

Zero --- FAIL lines. The package hit the wall: FAIL …/internal/vm 1801.594s against -timeout=30m.

The wall was the bug

-timeout on these lanes is not a hang detector. qemu-user is 50–100× slower than native, so it is the only thing deciding whether a slow-but-correct run counts, and it has to sit well clear of what the run actually takes. Measured on internal/vm, the heaviest package, in run 32661057000 — the verification run, all four green:

arch internal/vm
loong64 990s
riscv64 1204s
ppc64le 1284s
s390x 1754s — 46s under an 1800s wall

A 2.6% margin, which ordinary runner variance crosses — and the very next run of that same tree crossed it, at 1801.594s.

So the red is the wall and not the code. Nothing about the three fixes that landed today is implicated, and the three arches that were never near the wall stayed green in both runs.

60m puts s390x at about half the wall, which is what the comment above the line already claimed before the number was overtaken. The measurements go into that comment, so the next person does not have to take them again to know how much room there is.

Not done here, deliberately: sharding internal/vm across parallel jobs would cut the wall clock, and it is complexity this nightly safety net does not need to carry for a run that finishes comfortably inside an hour.

The nightly on main died with

    panic: test timed out after 30m0s
        running tests:
                TestStringToIBase (0s)

and no failing test: the s390x lane hit the 30-minute wall at 1801.594s.

The wall here is not a hang detector. qemu-user is 50-100x slower than
native, so it is the only thing deciding whether a slow-but-correct run
counts, and it has to sit well clear of what the run actually takes.
Measured on internal/vm, the heaviest package, in run 32661057000 — the
verification run, all four green:

    loong64   990s      riscv64  1204s
    ppc64le  1284s      s390x    1754s   <- 46s under an 1800s wall

A 2.6% margin, which ordinary runner variance crosses, and the very next
run of that same tree crossed it. So the red was the wall rather than the
code: nothing about the three fixes that landed today is implicated, and
the three arches that were never near the wall stayed green.

60m puts s390x at about half of it, which is what the comment above the
line already claimed. The numbers are in the comment now, so the next
person to look does not have to measure them again to know how much room
there is.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 6a0f9d0 into main Aug 24, 2026
7 checks passed
@tannevaled
tannevaled deleted the ci/arch-timeout-headroom branch August 24, 2026 08:49
tannevaled added a commit that referenced this pull request Aug 24, 2026
The lane installs qemu with

    sudo apt-get update && sudo apt-get install -y qemu-user-static

which is the exact line that sat for the better part of an hour in
go-crdt/crdt, on the same runner image, more than once. Bounding it there
identified the culprit: three attempts of ten minutes each produced not
one line of output from apt-get update, and the install was never
reached. The runner image already ships apt's lists, so the refresh was
not needed at all.

It has not hung here. But this lane runs once a night, takes forty
minutes when it works, and has just come back from seventeen consecutive
red nights — an hour lost to apt would read as another one, and nobody
would know which kind of red it was until they opened the log.

Same shape as the timeout in #436: what decides whether a slow-but-
correct run counts should not be something unrelated to the code.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant