Skip to content

Historical Rust pin Trial test times out on loaded macOS runners #1422

Closed

Description

@JacobStephens2

CI run 40438737527 on main (fff8895, merge of #1419) failed in `test-macos-shard (2)`:

`evals::experiment::execution::tests::trial_installs_its_historical_rust_pin_before_the_session` panicked at `src/evals/experiment/execution/tests.rs:250` with `Some("Trial timed out")`, for both Arms.

  • It is a flake, not a regression. The test last changed in c5310e7 (2026-10-26 09:17), four later main CI runs passed it on macOS, it passes locally in 0.9s, and re-running the failed jobs passed.
  • Cause, likely: `fixture.execution(5)` gives each Trial a 5s session budget. Both Arms run at once, and each session starts several `python3` processes (the session script, the `cargo` fixture, the `rustup` proxy path). This ran under nextest, alongside ~740 other tests on a macOS runner, and both Arms ran past 5s, around 7.9s.
  • Possible fix: give this test a larger timeout, since it checks install ordering, not timing. Or run the two Arms in sequence where concurrency isn't what the test is checking.

Chosen fix

Give this test a Trial timeout long enough for a loaded macOS runner, for example fixture.execution(30), and leave the two concurrent Arms as they are. The test checks that the pinned toolchain is installed once and before the session, not how long the session takes. Leave the other tests' timeouts alone. Done when the test still passes, and still fails if the install step is removed or runs twice.

Activity

  1. added
    needs-triageMaintainer needs to evaluate this issue
    ready-for-agentFully specified, ready for an AFK agent
    in-progressA Claim: thirdshift has taken this issue
    and removed
    needs-triageMaintainer needs to evaluate this issue
    ready-for-agentFully specified, ready for an AFK agent
    on Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions