Repository navigation
ci: an apt-get stall in engine-tests burns the 30-minute job budget and fails the required-check gate #974
Description
apt-get in the engine-tests install step has no timeout and no retries, so a
single mirror stall consumes the whole 30-minute job budget and surfaces as
cancelled — which correctly fails Required checks passed and blocks the PR.
Hit twice within forty minutes, on #971 and #972, both docs-only:
run 36887906077 Engines Python 3.13 (PCAP_CT) cancelled 15:57:45 .. 16:28:00 (30m15s)
run 36888646396 Engines Python 3.10 (PCAP_CT) cancelled 16:01:29 .. 16:31:47 (30m18s)
The log pins it precisely. apt-get update fetched InRelease metadata, then stalled:
15:58:06 Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
16:27:58 ##[error]The operation was canceled.
29m52s of silence, then the timeout-minutes: 30 at unit-tests.yml:327 killed the
job. Not version-specific (3.13 and 3.10) and not leg-specific: on the same run the
other five PCAP_CT legs finished in 39–81 seconds.
Nothing in either diff can affect an engine's system-package install, so this is
purely infrastructural — but it is expensive, because one stalled leg of six turns a
green PR red and the 30-minute burn delays everything behind it.
Fix is on the apt invocations at unit-tests.yml:403-417: -o Acquire::Retries=3 -o Acquire::http::Timeout=30 so a stalled mirror fails fast and retries, plus a
step-level timeout-minutes well under the job's 30 so a hang cannot eat the budget.
Worth noting Required checks passed behaved exactly right here — refusing to treat
cancelled as a pass is the property #966 and #970 added tests for.
Activity
JarryShaw commentedon Oct 1, 2026
Correction to this issue's own body, before #976 lands.
I recommended
-o Acquire::Retries=3 -o Acquire::http::Timeout=30. Per #976'smeasurement against apt 2.7.14 — the version noble ships — both are already apt's
defaults, so passing them would change nothing:
So my "fail fast and retry" framing was wrong twice. They were already in force —
which is itself the interesting part: it means whatever hung is not something apt's
data timeout governs. That value bounds each individual
select()wait rather thanthe transfer as a whole, so a stall that never times out a single wait can run
indefinitely. The 29m52s is consistent with that and not with a missing timeout.
Shipping the flags anyway would have added a comment claiming protection the step had
not gained — the "green it has not earned" pattern this repo keeps catching.
Two other things I got wrong here:
pypcap-parity's step atunit-tests.yml:640-642installs the union of the packagesengine-testsdoes,from the same mirrors, equally unbounded. Fixing only the leg that stalled would
have left it.
duration. Measured over 12 runs / 230 observations, the apt step's own p95 is
338s and its slowest successful run 508s on a PyShark cell. A 5-minute cap would
have failed ~4% of healthy legs — a worse defect than this one.
#976 therefore ships the step timeout alone, at 15 minutes, on both sites.
ci(unit-tests): bound the apt install steps so a mirror stall cannot …
ci(unit-tests): bound the apt install steps so a mirror stall cannot …
ci(unit-tests): bound the apt install steps so a mirror stall cannot …