Skip to content

Raise the harness timeout of the cargo_autoharness_fmt_impls test - #4923

Merged
feliperodri merged 1 commit into
model-checking:mainfrom
tautschnig:fmt-impls-timeout
Sep 30, 2026
Merged

feliperodri merged 1 commit into
model-checking:mainfrom
tautschnig:fmt-impls-timeout

Conversation

@tautschnig

Copy link
Copy Markdown
Member

Pass --harness-timeout 5m to script-based-pre/cargo_autoharness_fmt_impls, as the other autoharness tests (bounded, bounds, byte_str, c_str, formatter, wtf8) already do.

Why. On the macOS x86_64 runners (regression (macos-15-intel)), the LowerExp, UpperExp and Pointer harnesses of this test take 40–60 s, right at kani autoharness's default --harness-timeout of 60 s. When one of them times out, CBMC reports CBMC timed out instead of the expected failure, the test's Failed Checks: "lower exp" / "upper exp" / "pointer" line is missing, and the test fails. The same harnesses take about 10 s on Linux.

Evidence. I went through the logs of the regression (macos-15-intel) jobs between 2026-09-28 and 2026-09-30. 33 of them failed. This test failed in all 33, with 1–3 of those three harnesses timing out each time, and in 32 of them it was the only failing test. The one exception, #4903's merge-queue push run, also had cargo_autoharness_wtf8's len harness time out, at 5 m. On main pushes alone, the job failed 7 times out of 12 on 2026-09-30. The failures are independent of the changes under test:

The runner image version (20260819.586) was the same in passing and failing runs.

Manual testing. Locally the three harnesses take about 10 s each. Running the test with --harness-timeout 3s reproduces the CI failure exactly: the same three harnesses time out, their Failed Checks lines are missing, and the summary still reports 9 failures. With this change, cargo run -p compiletest -- --suite script-based-pre --mode exec cargo_autoharness_fmt_impls passes.

If the macOS Intel runner gets much slower, a longer timeout alone may not be enough; making these three harnesses cheaper would be the next step.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

On the macOS x86_64 CI runners (`regression (macos-15-intel)`), the
`LowerExp`, `UpperExp` and `Pointer` harnesses of this test take 40-60s, right
at `kani autoharness`'s default `--harness-timeout` of 60s. When one of them
times out, its expected `Failed Checks:` line is missing and the test fails.
In 33 failing macos-15-intel regression jobs between 2026-09-28 and
2026-09-30, this test failed every time, with 1-3 of those harnesses timing
out; in 32 of them it was the only failing test (in the other,
`cargo_autoharness_wtf8`'s `len` harness also timed out, at 5m). Locally
the three harnesses take about 10s, and a 3s timeout reproduces the failure
exactly.

Pass `--harness-timeout 5m`, as the other autoharness tests do.

Co-authored-by: Kiro <kiro-agent@users.noreply.github.com>
@tautschnig
tautschnig requested review from a team as code owners September 30, 2026 10:13
Copilot AI balanced review requested due to automatic review settings September 30, 2026 10:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The focused change matches existing autoharness test conventions and correctly addresses the observed timeout failures.

Review effort: Balanced
Findings: None

What changed in this PR

Raises the formatting autoharness test timeout to prevent macOS Intel CI flakes.

Changes:

  • Sets a five-minute harness timeout.
  • Enables the required unstable option and documents the rationale.
File Description
tests/​script-based-pre/​cargo_autoharness_fmt_impls/​fmt-impls.sh Configures and explains the increased timeout.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@feliperodri

Copy link
Copy Markdown
Member

let's merge and see if this decreases the noise. I'll keep an eye on those slow ones.

@feliperodri
feliperodri added this pull request to the merge queue Sep 30, 2026
@feliperodri feliperodri removed their assignment Sep 30, 2026
Merged via the queue into model-checking:main with commit 0ce5347 Sep 30, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants