Repository navigation
feat(stack): read eb_hooks.py limits and a memory cap in stack oom - #14
Merged
Merged
Conversation
The `stack oom` command warned on a non-integer divide_by_factor only in the hook rows copied into `data/eessi-bot-resources.toml`. The `--hooks FILE` option reads PARALLELISM_LIMITS from a software-layer-scripts eb_hooks.py with rustpython-parser, without running the file, and uses those rows in place of the table's. A key may be a module-level string constant such as CPU_TARGET_A64FX, and a repeated key keeps its last value, as in Python. A float factor is a warning that names the file and the line, since 16 // 2.5 is 6.0. These exit 1 with the line: an op other than divide_by_factor or set_maximum, a `** unpacking`, and a non-string key. The fixtures are `hooks/divide-2.5.py`, which warns, and `hooks/integers.py`, which does not and gives the bundled table's parallel levels. The `eb_stack_oom` tool takes hooks.
In software-layer-scripts eb_hooks.py, the post_ready_hook builds its log line in two statements. Since 8132905f the second is `msg+ "..."` on a line of its own. That computes the longer string and drops it, so the "to avoid out-of-memory failures" half of the message never prints. The `msg+` statement in eb_hooks.py drops its string; the fix belongs upstream. With `--hooks`, the `stack oom` command now walks the parsed file. It looks for an expression statement whose value is `name + ...`. It also looks inside def, class, loop, if, with, match and try bodies. It prints a warning with the file, the line, the statement and the enclosing function. The form `msg += "..."` is an augmented assignment and is not reported. The warning changes no parallel level and not the exit status. The `eb_stack_oom` tool returns these as hook_warnings. The fixtures `hooks/msg-plus.py` and `hooks/msg-plus-equals.py` are the `integers.py` rows with a post_ready_hook in each form. On upstream software-layer-scripts main (43df6f4) the warning is eb_hooks.py:440.
The memory cap of a package on a node is the level the hook leaves, the core cap, capped at the number of compiles of mib_per_job MiB that fit in the node's memory: `min(core cap, memory // mib_per_job)`. The `stack oom --node` command printed the hook's level and limit_by_memory in place of the hook, `max(1, min(cores, memory // mib_per_job))`, but not the hook's level with a memory ceiling on top. For a package with an mib_per_job, each `--node` line now carries "memory cap N (min(current, memory_mib // mib_per_job))" right after the current clause. The cap is not raised to 1: 0 is printed with "not one N MiB job fits", because one compile then needs more than the node has. Flooring the bytes to MiB first does not change the quotient, so the cap is also `memory_bytes // (mib_per_job * 2**20)`; a unit test checks that identity. Packages with no number print as before, so the bundled table, where libxc 7.0 is unmeasured, gives the same report. The `--mib-per-job FILE` option is a visible alias of `--job-memory FILE`, the table the cap reads. The Model Context Protocol tool keeps job_memory.
The post_ready_hook in software-layer-scripts eb_hooks.py does not divide the level EasyBuild chose. It calls `det_parallelism()` with no arguments, which counts the cores. Then the hook applies its PARALLELISM_LIMITS op to that count. It keeps the result only when it is below EasyBuild's level. EasyBuild 5.4.2 set_parallel uses `--parallel` in place of the core count when it is set, capped at `min(maxparallel, --max-parallel)`. The `stack oom` command modelled only the bot's case, where neither is set, so the difference never showed. The `stack oom --parallel N` and `--max-parallel N` options now give EasyBuild's level for the job: `--parallel` (which may be above the cores) or the cores, capped at max-parallel and maxparallel. The hook op still applies to the node's cores. On a 16-core zen4 node with `--parallel 12`, libxc gets `16 // 2 = 8`, not `12 // 2`. With `--parallel 4`, the hook's 8 is not lower and libxc stays at 4. The `--node` lines name the setting that decided (`then --parallel 4`, `then max-parallel 16`). The node line names `--parallel`. In a warning, the parallel line names `--parallel` and the cores the hook divided. It also says when the hook's result was not lower. A value of 0 for either flag is a usage error. The `eb_stack_oom` tool takes parallel and max_parallel. Without them the output is unchanged. At software-layer-scripts 43df6f4 the hook reads EESSI_SOFTWARE_SUBDIR, the software name, EasyBuild's level and `det_parallelism()`. No environment variable there caps PARALLELISM_LIMITS, so none is modelled.
The fixture `hooks/set-maximum-8.py` is `hooks/integers.py` with libxc's '*' row a cap of 8, `(set_maximum, 8)`, in place of `(divide_by_factor, 2)`: the form proposed in the review of software-layer-scripts pull request 337. With it, `stack oom --hooks` gives libxc 8 on the 16-core zen4 node, the same 8 as `16 // 2`, and pull request 1772 still warns on the 32 GiB zen4 node; only the parallel line changes, to "eb_hooks.py caps libxc at 8". A cap does not scale with the node, so the two rows differ on 64 cores (8 against 16 after max-parallel) and on 12 (8 against 6). The fixture `hooks/divide-8.py` has a reduction of 8, `(divide_by_factor, 8)`, as asked in the review of 337 (issuecomment-6599147327): `max(1, 16 // 8)` is 2, and pull request 1772 fits on both zen4 nodes. The how-to gets a section with the three commands. The eessi-extend skill says to check a scripts commit with `--hooks` before a software-layer pull request pins it in `bot/software_layer_scripts_commit`. The changelog has a Documentation entry.
The `--hooks` and `--parallel` sections of the out-of-memory how-to run `--node` before "A memory limit instead of a core fraction" says what it prints. The `--hooks` section links to that section, and the `--parallel` section names the `CORES:MEMORY_MIB:TARGET` form before its first example.
Bot job 221273 of software-layer pull request 1772 built libxc-7.0.0-GCC-15.2.0.eb on the 32 GiB Amazon Web Services zen4 node with `bot/software_layer_scripts_commit` set to a software-layer-scripts commit that divides libxc's parallelism by 3. Since `16 // 3` is 5, the log shows `make -j 5`, and cc1 was killed (issuecomment-6599075713). The table in `data/build-memory.toml` did not have that job, and with 4.0 GiB per process it predicted that 5 fits (20 GiB of 32). The observation is added, and the zen4 figure becomes the lower bound that explains it: 32 GiB over 5 is 6.4 GiB per process. The product `5 x 6.4` is exactly 32.0 in f64, so the table still reproduces the job. The two jobs at 8 ran out under both figures. A libxc warning on that node now reads `8 x 6.4 GiB = 51.2 GiB` and cites software-layer-scripts 337 as well. The maxparallel remedy is 4 in place of 7. The why line names 8 and then 5 compilers. The remedy line says the divider of 3 ran out too. It names a reduction of 8, 2 cores, from the review of 337 (issuecomment-6599147327). The review says to "really crank it" and "see if that works". The divider of 2.5 gives 6 processes or 38.4 GiB and no longer fits, so the float examples in the how-to and the tests now also warn about memory. The eessi-extend skill, the changelog, and the examples in the CLI reference and how-to follow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reproducer
From a checkout of this branch:
The
--node 16:32768:x86_64/amd/zen4value is one 16-core zen4 node with 32 GiB.On main,
stack oomhas no--hooks,--parallel,--max-parallelor--mib-per-joboption, so all six commands stop with a usage error.What and why
Software-layer pull request 1772 lost three zen4 bot runs to a killed
cc1while building libxc, two at 8 processes and one at 5.The run at 5 used a scripts commit that divides libxc by 3.
On main,
stack oomreads its hook rows from a copy indata/eessi-bot-resources.toml.Nobody can check a hook change with it before the change is merged.
--hooks FILEoption readsPARALLELISM_LIMITSfrom aneb_hooks.pyand uses those rows in place of the copy.The file is parsed with rustpython-parser and never run.
A float factor gives a warning with the file and line.
An op the model cannot follow exits 1 with the line.
msg+ "..."inpost_ready_hook.That half of the hook's log message never prints, and the fix belongs in software-layer-scripts.
--nodeline prints a memory cap beside the core cap for a package with anmib_per_job:min(current, memory_mib // mib_per_job).The
--mib-per-job FILEoption is an alias of--job-memory FILE.--parallel Nand--max-parallel Noptions model a job that sets EasyBuild's settings.The hook still divides the node's cores, not
--parallel.It keeps its result only when that result is lower.
data/build-memory.tomlrecords bot job 221273, which ran out atmake -j 5.The zen4 figure for libxc goes from 4.0 to 6.4 GiB per process, still a lower bound.
--hooks: a cap of 8,(set_maximum, 8), and a reduction of 8,(divide_by_factor, 8).Both forms come from the review of software-layer-scripts pull request 337.
The reduction of 8 is proposed there and not merged, and the current scripts divide libxc by 2.
Before and after
Each output here is quoted from the examples in
docs/orgmode/howto/oom-profile.orgon main or on this branch.The default run,
eb-stack stack oom --easystack $EASYSTACK, before:After:
Before,
resources-divide-2.5.tomlfitted on the 32 GiB zen4 node, at 6 processes of 4.0 GiB:After, it warns at 6 processes of 6.4 GiB, 38.4 GiB in all:
After, with
--hooksonhooks/divide-2.5.py:The dropped sum in
hooks/msg-plus.py, after:The 16-core zen4 node with
--parallel 12, after:The how-to also runs this node with
--mib-per-job tests/oom_profile_fixtures/job-memory-fixture.toml.That fixture gives libxc 8192 MiB as arithmetic, not as a measurement:
The cap of 8 in
hooks/set-maximum-8.pystill warns on the 32 GiB zen4 node, and only the parallel line changes:The reduction of 8 in
hooks/divide-8.pyleaves 2 cores, so libxc fits on both zen4 nodes:Tests
A cloud VM ran the checks on an earlier copy of the 7 commits and the docs build on the last one.
Three conflicts in the docs and the changelog were resolved after that run.
CI on this pull request is the run on these commits.
cargo fmt --all --checkcargo clippy --locked --all-targets -- -D warningscargo test --locked --all-targetspython3 scripts/gen-llms-txt.py --checksphinx-build -W --keep-goingOn the earlier copy, the 10 ignored tests need an EasyBuild tree or the network, and they're the same on every one of the 7 commits.
New tests cover the parsed hooks in
src/eb_hooks.rs, thestack oomcommand intests/oom_profile.rs, and theeb_stack_oomtool intests/mcp_v1.rs.Repro commands
The last line runs
scripts/sphinx-build-docs.sh, which callspython -m sphinx -W --keep-going.Note
Medium Risk
Changes OOM advisory output and libxc memory bounds that drive hook/remedy suggestions; new Python parsing could misread exotic hook files, but behavior is check-only and heavily tested.
Overview
stack oomgains ways to preview hook and EasyBuild parallelism before bot jobs run, plus sharper libxc zen4 warnings after PR 1772’s failure at 5 processes.--hooks FILEparses an EESSIeb_hooks.py(via neweb_hooks+ rustpython-parser) forPARALLELISM_LIMITSinstead of the bundled TOML copy; invalid rows exit 1 with a line number, non-integer divide factors and droppedmsg+sums warn.--parallel/--max-parallelmodel EasyBuild job settings while hooks still divide node cores and apply only when lower.--node(with--mib-per-jobalias) prints a memory capmin(hook level, memory // mib_per_job)besidelimit_by_memory.eb_stack_oomexposes the same flags andhook_warnings.data/build-memory.tomlraises libxc zen4 6.4 GiB/process (from 4.0), adds the job-221273 observation at parallel 5, and updates remedies (e.g. suggest maxparallel 4 and divide-by-8). Docs, CHANGELOG, and EESSI-extend skill describe the new flows and fixtures (set-maximum-8,divide-8, etc.).Reviewed by Cursor Bugbot for commit e435f29. Bugbot is set up for automated code reviews on this repo. Configure here.