You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Historical Rust pin Trial test times out on loaded macOS runners #1422
CI run 40438737527 on main (fff8895, merge of #1419) failed in `test-macos-shard (2)`:
`evals::experiment::execution::tests::trial_installs_its_historical_rust_pin_before_the_session` panicked at `src/evals/experiment/execution/tests.rs:250` with `Some("Trial timed out")`, for both Arms.
It is a flake, not a regression. The test last changed in c5310e7 (2026-10-26 09:17), four later main CI runs passed it on macOS, it passes locally in 0.9s, and re-running the failed jobs passed.
Cause, likely: `fixture.execution(5)` gives each Trial a 5s session budget. Both Arms run at once, and each session starts several `python3` processes (the session script, the `cargo` fixture, the `rustup` proxy path). This ran under nextest, alongside ~740 other tests on a macOS runner, and both Arms ran past 5s, around 7.9s.
Possible fix: give this test a larger timeout, since it checks install ordering, not timing. Or run the two Arms in sequence where concurrency isn't what the test is checking.
Chosen fix
Give this test a Trial timeout long enough for a loaded macOS runner, for example fixture.execution(30), and leave the two concurrent Arms as they are. The test checks that the pinned toolchain is installed once and before the session, not how long the session takes. Leave the other tests' timeouts alone. Done when the test still passes, and still fails if the install step is removed or runs twice.
Activity