Repository navigation
ci(python): contain and sample the experimental legs so a lost runner becomes a warning (#1740) - #1749
Merged
Conversation
… becomes a warning (#1740) GraalPy's leg loses its runner 18-30 minutes into its tests, which fails the job, and so its check, past any `continue-on-error`. experimental.yml: - `Start the resource monitor`: a detached sampler, outside the tests' scope, writes `free -m`, `df -h`, /proc/loadavg, the OOM-kill count, the scope's memory and the top five RSS processes every 30 s, each line led by `[monitor]`; `Run unit tests` streams it with `tail -F`; the stop step and the `resource-monitor-<version>` upload are `if: always()`. - `Run unit tests`: GraalPy alone gets GRAAL_PYTHON_VM_ARGS=--vm.Xmx2g and `-n 3`; pytest runs under `timeout --signal=INT --kill-after=3m 110m` and `prlimit --fsize=2147483648`, in a 12G, swapless, CPUWeight=20 systemd-run scope, probed with the same flags under a throwaway unit, else uncapped on the same argv. It writes the exit status, elapsed seconds, OOM kills and MemoryErrors to `tests-status`. - `Containment summary`, a new step before `Experimental summary`, warns of no scope, an OOM kill, a timeout (exit 124, or 137 with no OOM kill after 110 min) and a MemoryError, each with the last sample. `Experimental summary` is unchanged, the KTLO summary's script. - No cap, check name or job changes; nothing gates on it. Tests in tests/project/ pin and run each part; workflows.rst describes them. Refs #1740
JarryShaw
added
the
ci
Owner
Author
|
Verdict: GOOD TO GO at
|
JarryShaw
added
the
review: good-to-go
Contributor
|
Coverage: 89.57% (unit tier, Python 3.15,
Per-file detail: the |
JarryShaw
removed
the
review: good-to-go
3 of 13 tasks
JarryShaw
added a commit
that referenced
this pull request
Oct 15, 2026
) (#1755) #1749's 2 GiB heap holds, but a GraalPy worker's live heap grows with the tests it runs. In job 116172356218 the three workers reached 2.2-2.6 GB RSS in 16 minutes; four tests raised MemoryError, then gw1 raised one in pytest's own failure report, and the session ended in INTERNALERROR, exit 3. - Every GraalPy process gets --vm.XX:+ExitOnOutOfMemoryError with its heap: a spent worker exits, xdist fails the test it ran as crashed and starts a fresh worker. pytest gets --max-worker-restart=30, 30 restarts in all, where xdist's default is 4 a worker (12 at -n 3). - tests-status counts xdist's "node down" lines, and Containment summary warns "tests lost an xdist worker". The MemoryError count is word-bounded, so OutOfMemoryError and the flag itself are not counted. - The comments and workflows.rst give the measurements and their sources. test_graalpy_worker_heap_1753.py pins it: 7 of its 8 tests fail on main's experimental.yml, the eighth reads the docs. The 1740 and 1732 tests follow the new argv. tests/project: 921 passed, 1 skipped. Closes #1753
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please follow the guide below
You will be asked some questions, please read them carefully and answer honestly
Put an
xinto all the boxes [ ] relevant to your pull request (like that [x])Use Preview tab to see how your pull request will actually look like
Searched for similar pull requests
Followed the coding style (
make pylint,make mypy,make isort)make testpasses, and a test case covers the change. Narrow runs only, listed below.Added a changelog entry under
docs/source/changelog/and regeneratedCHANGELOG.md, if the change is user-visible: N/A — centralised in docs(changelog): shared 1.5.0 changelog — long-lived, merges last (#610, #616, #617, #618, #620) #657What is the purpose of your pull request?
Tick the commit type your subject line carries.
fix— corrects a defectfeat— adds a featureperf— changes performance, not behaviourrefactor— changes neither behaviour nor performancetest— tests onlydocs— documentation onlyci— workflows or build toolingrelease— bumps the version or rolls up a distributionchore— anything elseRefs #1740. The owner's rule is that a non-blocking leg may fail only as a warning, and its check must stay green. GraalPy's leg keeps losing its runner 18–30 minutes into its tests, and a lost runner fails the check no matter what
continue-on-errorsays. This keeps the runner alive, so a resource breach fails the step instead, and that is already a warning.[monitor]lines in the tests' log, uploadedif: always().--vm.Xmx2gand-n 3; 3.14t keepsauto.timeout --signal=INT --kill-after=3m 110mandprlimit --fsize=2 GiB, inside a 12 GB, swapless,CPUWeight=20systemd-run --scope.Containment summary, a new step, warns separately on no cap, an OOM kill, a timeout (exit 124, or 137 after 110 minutes) and a MemoryError, each with the last sample.Experimental summaryis byte-identical tomain's.-n 3, GraalPy likely needs about 117–151 minutes (an inference), so it may end on the 110-minute timeout as a warning. That is expected and non-blocking. Caps, check names and jobs are unchanged.tests/project/test_resource_monitor_1740.py, withsudoalways stubbed. 1672's list of steps that carryLEGgainsContainment summary.sudoplussystemd-run, the cgroup path, and whether the sizes keep the runner alive. This PR's own Experimental Tests run is the first evidence.