Repository navigation
Compile OOM defense: cap, yield and shed compiles, kill them first, requeue what was killed - #1781
Merged
Merged
Conversation
christianparpart
force-pushed
the
feature/compile-oom-defense
branch
6 times, most recently
from
November 15, 2026 23:27
54f28b8 to
c35bc1f
Compare
A leader re-sends every entry a follower has not acknowledged on every heartbeat (50 ms). The follower's in-memory log already skipped the ones it held, but `RaftNode::OnAppendEntries` persisted from `prevLogIndex + 1` on any non-empty request, so each re-send truncated the durable log and rewrote it: two flushes per heartbeat. Once a flush outlasted half the interval the follower fell behind the re-sends for good -- its reactor stayed in `SaveLog`, its acknowledgement never reached the leader, and a joining voter was never promoted. It also cut acknowledged entries on disk for the length of a rewrite, the hazard `TryAppend` documents and avoids in memory. `RaftLog::AppendOutcome` now says where the log CHANGED (`changedFrom`, also the conflict's truncation point, nullopt for a run already held or a rejected append), and the follower persists and re-derives its configuration from there only. Measured: "A two-voter fleet's follower holds its leader as a voter ..." was red on Windows CI three times (2 members, 1 voter, stalled ~9.5 s), and none of 2,368 local runs under CPU overcommit reproduced it (two unrelated port collisions). With fsync slowed by an LD_PRELOAD shim (gcc-release): 40/40 failed at 20-150 ms and 20/20 at 20-60 ms, with the CI's exact message; 10-40 ms passed 20/20. After: 200/200 at 20-150 ms, 200/200 at 20-60 ms, 40/40 at 100-300 ms, and on clang-debug 200/200 (shim) plus 240/240 (plain). With the old persistence call restored the three new RaftNode cases fail, and only they; the conflict at prevLogIndex + 1 stays green as the control. What remains is #1782: a follower still persists synchronously on the reactor that reads its leader's session, so a long run of genuinely new entries on a disk whose flushes outlast the heartbeat can still keep that reactor from sending the acknowledgement. Signed-off-by: Christian Parpart <christian@parpart.family>
…t that includes it
The clang-tidy job's "Analyse the changed lines" step hands every changed
header to clang-tidy-diff.py, which runs `clang-tidy -p=<db> <header>`. A
header has no compile_commands.json entry, so clang-tidy interpolates a
command from the entry whose PATH looks most like the header's, which knows
nothing about includes. For a test-only header that guess is often a unit
built without the include directories the header needs, and the run fails
`[clang-diagnostic-error] ... file not found` on a header that compiles in
every unit that includes it.
CI run 43735465655 (job 132877030688) failed that way on four headers:
src/tests/CgroupFakes.hpp ('Pace/Platform/CgroupV2.hpp' not found),
src/apps/pace-cc/OomVictimTestSupport.hpp, src/apps/pace-node/
AdmissionTestUtils.hpp and src/apps/pace-node/WorkerTierTestFixture.hpp
(catch2 not found). CgroupFakes.hpp is guessed wrong from master's own files
alone, so this predates the branch that found it. Reproduced locally with
clang-tidy 22.1.8 against the gate's clang-debug database.
scripts/tidy-header-commands.py writes a NEW database under $RUNNER_TEMP
(tidy-db), never the build tree's, so ninja's generated-header pass and
anything else reading the build tree see only what configure wrote. Every
input entry is copied unchanged; each selected header gets one entry
borrowed from an includer: the includer's directory and argument vector,
with the unit's source, `-c` and the `-o <object>` pair removed and
`-x c++-header <header>` appended (c++-header, so a header as a main file
draws no `#pragma once in main file`). The step points clang-tidy-diff.py's
-path at that database and keeps the `Failed:` grep and PIPESTATUS check.
The includer is found by RESOLVING each `#include` against the including
file's directory and every -I/-isystem/-iquote/-idirafter directory the
database names, following only files under src/. This deviates from the
suffix matching tidy-sweep.sh's BuildIncludeGraph uses: resolution needs no
listing of the tree (so no third-party-roots enumerator), and the files are
exactly the units the database names and what they reach. A breadth-first
walk over the reverse graph takes the first depth holding a database unit,
so a direct includer beats a transitive one; ties prefer the unit whose stem
is the header's (Foo.hpp -> Foo.cpp / Foo_test.cpp), then the smallest path.
A header no unit includes gets no entry, so clang-tidy still interpolates
its command -- but over the AUGMENTED database, whose borrowed header entries
are candidates too, so the guess is not necessarily the one it made before.
Such a header is now NAMED in the step's output; it is never dropped from
the analysis. The header extensions are written once in the step: one case
arm both selects a header and collects it for the script.
Limits. GNU-style driver lines only: the Linux leg calls this,
clang-tidy-windows does not, and the self-test is registered `if(NOT WIN32)`.
The include directories searched are the UNION over every unit, so a bare
quoted spelling can resolve through another target's directory and name a
false includer whose command lacks the header's own -I. Paths are compared
after abspath/normpath, never realpath, so a work tree reached through a
symlink fails safe: every header is named as having no includer and keeps
the interpolated command.
Measured: the four headers get the commands of CgroupV2_test.cpp,
OomVictimExec_test.cpp, AdmissionResponder_test.cpp and
NodeConditions_test.cpp. clang-tidy-diff.py, run as the step runs it on
`git diff -U0 origin/master...HEAD` of the four, exits 1 with the
clang-diagnostic-error on the build tree's database and 0 with no finding on
the augmented one. A probe database adding `-include/nonexistent-probe.h` to
each borrowed entry made clang-tidy fail on that include for all four, so the
borrowed entries are the ones used rather than a guess that happens to work.
Over all 40 headers the branch changes, all 40 found an includer and
clang-tidy-diff.py reports nothing.
The tidy-header-commands-selftest ctest (hygiene, default set, not on
Windows) drives a synthetic tree and database built with `arguments`, so a
scratch path containing a backslash passes 10/10: a direct includer's
command with the source swapped, the same-stem unit preferred although it
is the LARGER path, a direct includer (Z_user.cpp) beating a deeper unit
that would win the tie-break, a header reachable only transitively, a
header with no includer getting no entry and being named, the existing
units unchanged. Neutering each failed exactly the cases covering it:
the stem key replaced by `key=lambda unit: unit` (1), collecting every
reachable unit before the tie-break (1), the source swap (1), the naming
(1), and the transitive walk (3).
Signed-off-by: Christian Parpart <christian@parpart.family>
…ds the tests gcc 16.2 at -O3 refuses McpNodeTools_test.cpp under -Wnull-dereference: `*values.front().AsString()` dereferences a pointer the optimiser cannot prove non-null, the same shape 5e17981 converted elsewhere. Two more files carry it unguarded (McpToolchainTools_test.cpp, McpActionTools_test.cpp); they are converted with it, so the next gcc does not find them one at a time. Signed-off-by: Christian Parpart <christian@parpart.family>
…thread
SampleOnce documents itself as "take one sample now, whatever the timer is
doing", and nineteen test calls drive it while the thread the constructor
starts samples too. A sample reads its sources and only then records, and a
history keeps its newest record as the reading the next delta is taken from.
Two unserialised samples could therefore record out of the order they read in.
Measured with traces: the thread read DispatchGranted=0 before the case's
queued grant and recorded it after the case's own sample had recorded 1.
That overwrote the t1 bucket's raw value and `_last` with the stale 0, so
"A compile the admission queue placed on a worker reads as dispatched on the
fleet page" read "dispatched":{"value":0. Production has one caller (the
thread), so no shipped behaviour changes; the seam's promise was false.
The fix holds one mutex across a whole sample: every read, every record and
the keyed histories' fold.
New case "Two samples taken at once never overlap so a stale reading cannot
be recorded last" asks the property directly. The injected node-facts
provider, called inside a sample before it records, counts samples inside
at once. The first entrant waits (bounded, 500 ms) for a second, while the
case and a helper thread both sample; the peak is asserted after the join.
Lock removed: ran=100 fail=100 idle and ran=100 fail=100 under load.
Lock present: ran=100 fail=0 idle and ran=100 fail=0 under load.
Rates for the dispatched case (scripts/flake-rate.sh, clang-debug, 32 busy
loops on 32 cores):
before: ran=150 pass=143 fail=7 (idle host: ran=50 fail=0)
after: ran=150 pass=150 fail=0 (idle host: ran=100 fail=0)
Signed-off-by: Christian Parpart <christian@parpart.family>
A node told to stop while its toolchain walk was hashing skipped the rest of the files in AnnouncingParallelFor and still answered IParallelFor::Run with "every slice ran". ProbeToolchainFiles therefore reported the scan complete, and CachedToolchainFingerprint wrote the short digest into the machine-wide fingerprint cache under a stamp that validates forever -- the stamp folds each root's path and mtime, never its contents. Every later node on that machine then served a fingerprint no client computes. dist-compile-e2e reads its worker's first `serving ... as` line while isolating only its launcher's state, so it failed on the mismatch and, by tearing its worker down mid-walk, poisoned the entry again for the next run. The code is unchanged from master; master fails the same way against the same cache state. A skipped slice now makes Run answer false, so the walk is incomplete and nothing is cached, as IParallelFor documents for a slice that did not finish. The new case stops the survey from inside the include probe, so the stop lands in a walk that has begun -- the existing stop case uses a token stopped beforehand and never reached the walk. Signed-off-by: Christian Parpart <christian@parpart.family>
…s own dist-compile-e2e isolated its launcher's state but not its nodes'. A worker fingerprints its toolchains through the launcher's code into the launcher's machine-wide cache, so the fixture's worker served whatever the developer's ~/.local/state/pace-cc/toolchains held and wrote into it: one truncated entry there failed the fixture run after run and read as a branch regression (#1774). The node now gets the launcher's seam. e2e_node_state_enter writes a shim through the same writer and the same StateDirectoryImpl rows as e2e_launcher_state_enter, over <workdir>/node-state, and points the node variable at it; New-E2ENodeState / Use-E2ENodeState do the same around one child in PowerShell. The root is deliberately NOT the launcher's: with one root the launcher would read the worker's entry instead of walking, and the fixture's launcher/worker agreement check would compare one answer with itself. The positive control is the worker's walk caching HERE (e2e_node_state_assert_used --fingerprinted, Get-E2ENodeFingerprintCount). Every fixture whose node runs a worker, and so a toolchain survey, is converted: dist-compile-e2e.sh, fleet-dashboard-e2e.sh, node-socket-activation-e2e.sh, dist-compile-e2e.ps1 and node-scratch-isolation-e2e.ps1. node-mcp-e2e.sh already isolated its workers by hand; the rest start only --slots=0 nodes or nodes refused at startup, which survey nothing. rpm-package-e2e.sh and macos-package-e2e.sh are out of scope: they install the package and run the node as the packaged service, inside a throwaway container or under the system service account, so the state directory it caches into is that container's or that account's, never the developer's. check-node-fixture-starts now enforces the seam: a `runs` start naming no --slots=0 and no inline XDG_STATE_HOME= is reported `unisolated-node <file>:<line>` unless a .sh entered e2e_node_state_enter above it or a .ps1 start sits inside Use-E2ENodeState. exit-codes-e2e and node-config-file-e2e are exempt by row: their starts are refused by the startup rules before any tier is built. e2e-helpers-selftest drives the control's pass, foreign-root, zero-run, empty-record and not-walked verdicts through the real shim. The shim redirects HOME as well (LOCALAPPDATA on Windows) for the node and every compiler it runs, which reaches further than the fingerprint cache; the library and the rule say so. Refs #1774 Signed-off-by: Christian Parpart <christian@parpart.family>
…wrote 297a4e0f stopped a node from caching the digest of a walk a stop cut short, but it reached only NEW entries. One already written keeps validating, because the stamp folds each root's path and mtime and never its contents, and nothing in an entry tells a truncated digest from a whole one -- so every node and launcher on such a machine kept serving a fingerprint no client computes until something forced a full walk. The stamp's own schema moves from toolchain-stamp-v2 to v3, so every entry an older build wrote is a miss: rewalked once and written back. The fingerprint VALUE is untouched, so a fleet mid-upgrade still matches; the cost is one walk per toolchain per machine. The schema is now a public ToolchainStampSchema and a defaulted parameter of ComputeToolchainStamp, as FingerprintSchema already was, so a test can stamp an entry the way the older build did. The new case plants a wrong fingerprint under a v2 stamp beside a control planted under v3, which is served. With the constant put back to v2 (and its guard REQUIRE removed) the case's four behavioural checks fail and nothing else in [toolchain-probe] does. Closes #1774 Signed-off-by: Christian Parpart <christian@parpart.family>
The verb table showed LEASE as [fingerprint][objectKey][codecs], which was already stale before this branch: EncodeLease sends six fields -- fingerprint, objectKey, codecs, exclusions, toolchainLabel, origin. The table now shows all six and a paragraph says how each is encoded: codecs one byte per id, the exclusions a nested list of at most 16 endpoints (MaxLeaseExclusions) of 1 to 261 bytes, refused rather than truncated past that, the label UTF-8, and the origin one byte below AskOrigin::Last. Signed-off-by: Christian Parpart <christian@parpart.family>
Signed-off-by: Christian Parpart <christian@parpart.family>
Signed-off-by: Christian Parpart <christian@parpart.family>
A compiler the worker's machine killed -- the compile memory ceiling, the
node's own shedding, or any other SIGKILL -- answered nothing about the
code, yet reached the client as a CompileOutcome with exit 137: a compiler
failure in a TU that compiles. It is now JobRefusal::CompileKilled, answered
worker-compile-killed (0x35, a new byte and no version bump: the reply
grammar is unchanged and an older launcher reads an unknown refusal and
compiles locally all the same), graded WorkerRefused by the launcher.
One CompileKillTable names each cause (ceiling, shed, killed) and its
counter: pace_worker_compiles_killed_{ceiling,shed,signal}_total. The node
logs one Info line per kill naming the cause and the source
(KilledCompilesLog, a decorator in front of RefusedArgumentsReport).
The worker reports NO peak for a killed compile: it was cut short, so its
peak is a lower bound, and the client's local recompile settles the
estimate with the true figure. This refines the spec's call C5
(max(known, observed)) and must be called out in the PR.
The live-stats layout pin moves with the three counters.
Signed-off-by: Christian Parpart <christian@parpart.family>
…able Signed-off-by: Christian Parpart <christian@parpart.family>
The worker now writes compiles/memory.max at startup -- unconditionally, so a compiles/ an earlier run left behind does not keep that run's figure -- and again after every accepted reload. The ceiling is the new --compile-memory-limit=<size|N%> (YAML compile_memory_limit, reloadable, local wiring), else the machine budget MachineBudgetOf reports, so admission and the cap are one number (CompileCeilingOf). Typed below that budget it lowers the reported budget too (C3). A worker whose ceiling cannot be held raises the new latched compile-memory-uncapped condition beside compile-memory-unmeasured, with the preparation's (or the write's) failure as its detail; it is answered only by PutCompileCeilingInForce, so a wiring that forgets the ceiling reads undecided. The condition table spent its spare row: MaxNodeConditions 26, bought with one more KiB of HistoryPayloadShare, no field ceiling moved. PrepareJobCgroups no longer swallows every refusal on compiles/'s memory.swap.max and memory.oom.group: only an absent or unsupported file is tolerated, listed on CgroupLayout::notApplied and said in one Warn line at startup; anything else is CompilesNotCreated. A job's memory.oom.group refused for another reason fails Create and removes the directory. Signed-off-by: Christian Parpart <christian@parpart.family>
…y of its own compile-oom-e2e.sh started a worker with the caller's state directory, so its toolchain walk wrote the developer's fingerprint cache, and `node-fixture-starts` refused the tree: it is the one fixture #1774's isolation seam did not reach. Entered after the launcher's, as every other fixture that starts a worker does. Signed-off-by: Christian Parpart <christian@parpart.family>
…t fits A compile asked for again after its compiler was killed for memory says so on its ADMIT: `AskOrigin::AfterKill`, a third value of the byte ADMIT and LEASE already carry. The kill itself reaches the leader the way it already did -- the launcher ends the admission unsettled, and `Abandon` raises the key's estimate (`RecordKilled`) -- so the requeue is charged that. The queue then places such a request only on a machine whose CONFIGURED budget holds its estimate. A smaller one is no target at all, where a first request would run alone on it: there the compile ceiling is the budget, and the run would be the ceiling's next kill. A budget the pressure allowance holds down is waited for, never ruled out. When every reported machine the request could run on is too small, its caller included, it is decided `neverFits` and granted `Local` with the new `LocalGrantReason::NeverFits`, alone on its caller -- the client runs it outside the ceiling. Both enumerators land in wire version 18, which is not yet released: the precedent of its MACHINE-BUDGET fields, and a version-17 decoder refuses either as past its `Last`. Counted by two catalogue rows the admission surface writes, `pace_admission_requeued_after_kill_total` and `pace_admission_never_fits_total`; the live-stats layout pin and the attribution figures move. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…n it The seam for the compiles `pace-cc` runs itself. A node names its `compiles/` cgroup in a rendezvous file in its runtime directory (`RendezvousFile`, written atomically, quiet on repetition); a launcher reads `$XDG_RUNTIME_DIR/pace-node/compile-ceiling` (`RendezvousLocator`), which is that file exactly under the user unit, the one place the two share an owner. A node with no `RUNTIME_DIRECTORY` names nothing, so a node started by hand can never overwrite or withdraw the rendezvous of the node the user runs. `LocalCompileRunner` places a compile in a cgroup of its own below that ceiling (`CgroupJobSource`, now with a per-owner name prefix so a launcher's `pace-cc-<pid>-<n>` never takes a node's `job-<n>` and the node's startup sweep never takes a launcher's), and runs it under the OOM-victim shim wherever there is one. Which applied is a named outcome per compile -- `LocalDefense` and, when not the ceiling, the `CeilingMiss` that says why -- read off whether the child actually started in the cgroup (`CompileRun::inCgroup`, set by the runner) and off what the cgroup source heard, never assumed. `CgroupV2.cpp` (std-only) is compiled into the launcher for it, and so is the shim, which the launcher now spawns through. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…s under the ceiling A worker's `worker-compile-killed` no longer compiles here at once, outside every layer of the defense. It is its own decline cause (`MemoryKilled`, `IsWorkerCompileKilled`), and the miss path ends that admission UNSETTLED -- which is how the leader learns a kill -- and asks once more as `AskOrigin::AfterKill`, so the queue places it where its estimate fits, now or once room frees. One requeue per compile, whoever killed it (`KillStep`, `RunMiss`): - a second worker kill compiles here, asked for local-only with that worker excluded, tallied `killed for memory twice, so compiled locally`; - a `NeverFits` grant runs here OUTSIDE the ceiling, tallied apart; - a compile the local ceiling kills waits in the queue once in the same way, and killed there again runs once more outside the ceiling under the same grant; - a kill outside every ceiling ends the compile killed, said on stderr whatever the verbosity, and teaches the leader nothing of a lower-bound peak. With no queue to wait in, a compile the ceiling killed runs once more outside it at once. Every compile the launcher runs itself goes through `LocalCompileRunner`, so it joins the local node's ceiling where it can and starts as the first OOM victim everywhere else -- the launcher is its own `pace-oom-victim-exec` shim -- and the invocation log records which in a new `local-defense` column (format v4; a v3 line is read as its prefix), which `-s` counts. The real compile-OOM fixture asserts the requeue: on its one 256m machine the kill is followed by `asking the queue once more`, and then never-fits or a second kill. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…self While its ceiling holds, a node writes its `compiles/` path to `compile-ceiling` in its runtime directory, and withdraws it when the ceiling does not hold -- from `PutCompileCeilingInForce`, the one place a ceiling is put in force, through the keeper every later write goes through. The rendezvous rides the process's one `CompileMemoryMeter`, so a formation reform keeps it. A failure to write it is said at Info once; the ceiling works without it. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…de's ceiling `pace-node.md` gains the two sections the behaviour needs -- a killed compile waits for memory, and the compiles pace-cc runs itself -- and its kill sentences stop saying the client compiles locally at once. `pace-cc.md` names the three new tally rows, the `local-defense` column and its format version, and the two queue rules. The rulebook states both as rules of the compile queue and drops #1770 from its open work. Closes #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
A runner given a cgroup source answered no peak at all for a child that ran outside its cgroup -- no cgroup could be made, or the system would not start the child in one. That is right for a node, whose `ru_maxrss` includes its own resident set, and wrong for the launcher since #1770 gave it a cgroup source: every local compile in an ssh session's scope, or under a node whose `compiles/` refused this user, settled with no peak, and the estimator stopped learning what it learned before the ceiling existed. The fallback is now the runner's own peak source (`MakeProcessRunner(cgroups, shim, outsideCgroup)`): `Unmeasured` by default and for a node, `ResourceUsage` for the launcher. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…hold After the second worker kill the miss path asks local-only, that worker excluded -- and asked it as `AfterKill`. For a local-only request the leader judges never-fits against the caller ALONE, so a 6 GiB laptop whose 32 GiB workers had both held the compile was answered `NeverFits`: the leader counted `pace_admission_never_fits_total`, and the client's grant seam overwrote its `killed for memory twice` tally with *no machine's budget fits it*. The local-only fallback after a failed or killed worker now asks `AfterGrant` (`Waits::LocalOrigin`): the same compile asked again, which is what it is. `AfterKill` stays the one requeue's, judged against every machine the request could run on. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
The requeue after a kill was charged `max(known, default)` -- for a key never measured, exactly what its first ask was charged -- so a compile that needed more was placed on an equal machine, killed again and compiled locally: "wait until space is free" was "try once more". The kill now tells the leader what the compile held when it died: - a worker's `worker-compile-killed` reply carries it as an error TRAILER -- eight big-endian bytes after a `0x00` that ends the message (`EncodeKilledPeakTrailer`, `ErrorTrailerOf`, `KilledPeakOf`), so every reader of the message reads exactly what it read before; - the client settles the admission `SettleOutcome::Killed` with it, and with its own cgroup's peak for a compile killed here, instead of ending it unsettled; - the leader folds it in as a LOWER bound (`PeakEstimates::RecordKilled`: raise, never replace, never a measurement), so the requeue is charged at least what it held and placed only where a configured budget holds that. The trailer and the outcome land in unreleased wire version 18, as the rest of #1770 does. `wire-tag-reach-count` moves to 312 for the trailer's case. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…he ceiling waits once too `pace-node.md` said a compile killed outside every ceiling is not retried; its FIRST kill is requeued once like any other, and only a second one ends the compile. The kill sections now say the admission is settled `Killed` with what the compile held rather than ended unsettled, that the fallback after a second kill is never counted never-fits, and that a compile outside the ceiling is still measured. The rulebook bullet says the same. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…it at once Since the fallback after a second kill asks `AfterGrant`, its grant came back `Local` with reason `Asked` and the launcher joined this machine's ceiling: with the estimate the kills taught above the machine's budget -- which IS its ceiling -- that run was killed a third time before `RunHeld` tried outside. A wasted compile, and the machine driven to its ceiling, on every killed-twice TU. The leader now says why such a grant runs alone: `LocalGrantReason::RunsAlone` (v18, unreleased), for a `Local` decision that ran alone and is no never-fits -- counted as nothing beyond the `AdmissionRanAlone` it already is, so never-fits stays at zero. The launcher's `LocalGrantTable` gains two placement columns, before and after a kill (`PlacementForGrant`): `RunsAlone` keeps the ceiling for a first ask, whose estimate is a guess, and runs outside it once a kill taught the estimate; `NeverFits` runs outside always. Also: the comments that still said a kill ends the admission unsettled, and `EncodeErrorReply`'s Doxygen block, which sat above `ErrorTrailerSeparator`. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
What a killed compile held is a figure a peer states, and `RecordKilled` raises the key's estimate and its source shape's maximum for good: one forged trailer could make every configuration of that source run alone. A compile cannot hold more than the machine it ran on lets its compiles hold -- its ceiling is that budget -- so `AdmissionQueue::HeldBound` clamps the figure to the budget that machine last reported. The charged-at-least case is reworked to a kill within that bound. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…ys otherwise `--compile-memory-limit`'s help, the reference configuration's two mirrors of it and of the yield, the worker's kill log line and the yield row of `pace-node.md` still said a killed compile's client compiles it locally. It asks the queue for it once more, charged what it held. `pace-node.md` also says when a killed-twice compile runs outside its machine's ceiling, and what bounds a kill's figure. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
… request Comment and reference-configuration text only, no behaviour change: `memory_pressure_shed` said a shed compile's client compiles it locally, where it asks the queue once more; `AdmitGrant::reason` said only a dispatchable request is given a reason, where `NeverFits` and `RunsAlone` answer `LocalOnly` too; `LocalPlacement::OutsideCeiling` did not name a `RunsAlone` grant after a kill; a stray blank line sat between `EncodeErrorReply`'s Doxygen and its declaration; and the `compile_memory_limit` comment ran past the file's width. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
… v18 ADMIT/SETTLE values The `0x35` worker-compile-killed row still said the client compiles locally and the worker reports no peak. It now names the trailer (0x00, then the peak as an 8-byte big-endian u64), the SETTLE `Killed` outcome carrying it, and the one `AfterKill` requeue, with local fallback only for never-fits or a second kill, all inside wire version 18. ADMIT gains its `origin` byte, the grant reasons `4` never-fits and `5` runs-alone, and SETTLE outcome `4` killed. Refs #1770 Signed-off-by: Christian Parpart <christian@parpart.family>
…, one at a time dist-compile-e2e hung in case 9 on a host whose memory PSI was high for a reason no compile caused: another cgroup thrashing at its own memory.max. The cache-only node (--slots=0), leader of its own one-machine fleet, reported Shed in its MACHINE-BUDGET, and its launcher's ADMIT waited for as long as the pressure lasted -- here, forever (`admit -> no-reply`). Introduced by 2846351d (feat(distributed): admission holds back a machine under memory pressure), which made AdmissionQueue::Pump's grant predicate answer `false` for every target reporting a non-Calm level, the caller's own idle machine included; e984ab4e widened it from Shed to Yield, and eea25fba made it reachable, since from then every node reports the level the host's PSI implies, even with no compiles cgroup to act on. The queue promises a miss is never starved (#1728); on an idle machine waiting relieves nothing the fleet holds, and nothing it does can end it. A pressed machine now takes nothing new while any compile of the fleet runs or is reserved there, as before, and no other machine's work at all; idle, it runs its OWN next compile that fits the budget it reports -- the allowance, which 2d24e1ba already floors at one default estimate so that exactly one compile fits. A compile larger than that still waits, as the closed run-alone exception requires. The shared idle conjunct is one `idle` for both that floor and run-alone. Tests: AdmissionQueue gains the floor (Yield and Shed, one at a time), its scope (no other machine's work, nothing oversize), and the shed case now holds a ticket beside a RUNNING compile; AdmissionResponder pins both at the wire. Neutering each conjunct of the floor turns red exactly the cases that guard it. The docs, the wire header and the governor's log line say "beside a running compile". Signed-off-by: Christian Parpart <christian@parpart.family>
… machine's The governor read root PSI as machine pressure, and a service stalling at its own memory.max raises that as surely as a shortage does -- with GiBs free. On the host dist-compile-e2e ran on, one user service doing exactly that held every node at Shed (system full 14.8%), which freeing memory cannot relieve and which yielding compiles cannot end. The source now finds such cgroups (Platform::ConfinedStallScan, through the ICgroupHost seam): a cgroup outside this node's own subtree, its ancestors and the compiles', that carries 80% of the machine's some or full stall while standing at its own memory.max or memory.high (an OwnLimitTable row each) -- memory.events count risen since the previous sample, or memory.current within a twentieth of the limit. The governor counts such a stall the way it counts the compiles' own: it neither yields nor sheds and eases a level, with at-limit evidence held two dwells per cgroup. One cgroup must carry the share alone, so a real shortage, which stalls the rest of the machine too, still sheds. Cost and staleness: the tree is walked only while the machine stalls at all, at most once per 10 s, descending only into a cgroup carrying the share (PSI covers the subtree), capped at 64 cgroups and 8 levels; each sample re-reads at most four kept candidates. A cgroup reaching its limit after a walk counts as the machine's until the next one -- a yield it did not need, never a freeze missed. Control files are told from child cgroups by the directory's own cgroup.controllers. An unlistable tree is no confined stall at all and is said once at Warn (and once at Info when readable again); the discounted cgroup is named at Info when the discount starts or ends, never per sample. Neutering the shed-entry discount, the yield discount, the evidence lapse, the own exclusion, the at-limit requirement, the pruning, the walk interval, the calm guard and both watcher transitions each turns red exactly the cases guarding it. Signed-off-by: Christian Parpart <christian@parpart.family>
memory.events is hierarchical: a child reaching its own limit counts in every ancestor's file too, so a slice with a numeric limit far above what it holds read as "at its limit" whenever a child of it was. The confined-stall scan now reads memory.events.local where the kernel offers it (5.8 and later), the hierarchical file only where it does not. Neutering the order turns red the one case that guards it. Signed-off-by: Christian Parpart <christian@parpart.family>
…it, and a settle frees what a report counted Two ways e392c45d's floor -- an idle pressed machine still runs its own compiles, one at a time -- was still starved, from its review: - Another machine's dispatchable request whose candidates named a pressed worker became that worker's head and, a pump later, reserved its whole budget there. The hold kept the worker from ever being idle, so its own launcher waited as long as the other caller stayed blocked: case 9's hang, moved to a fleet. A request that can never run on a pressed machine -- another machine's work, or one above the budget it reports -- is now neither its head nor reserves there (BarredWhilePressed, which the grant check reads too, so the rule has one definition). - The ledger counts max(charged, reported held). A report that arrived while a granted compile ran counted it, and after the settle kept the machine non-idle until the next report: the floor ran one compile per 20 s. A settle now gives back what a report received AFTER its grant counted (Charge::reportsAtGrant); one taken before the grant owes nothing. A report taken before a grant but arriving after it would lose another compile's count instead -- fail-open for one report period at most, against fail-closed for every one. Tests: the floor's second request is now held by `idle` alone (1 GiB on a 2 GiB budget); a machine reporting no slot runs nothing; the shed case pins charged, report-only held slots and report-only held bytes apart; the reservation and give-back cases with a control for the report order. Neutering the skip, the slot conjunct, each used-* conjunct, the order condition and the slot give-back each turns red exactly its case. The hold.* half of `idle` stays green under neutering: after this change a pressed machine's holds come only from its own fitting head, which is blocked only while the machine is not idle anyway. Refs e392c45d Signed-off-by: Christian Parpart <christian@parpart.family>
Under Shed the governor killed the youngest compile once per dwell whenever compiles held anything. With e392c45d's floor that is the one own compile a pressed machine still runs: killed, requeued AfterKill, admitted again, killed again, and run OUTSIDE every ceiling after the second kill (MissFlow) -- every own TU compiled three times during a real freeze, ending with no memory bound at all. The owner's ruling, as a table: what a shed does about the running compiles is a ShedKill verdict (ShedKillTable, RowsInEnumeratorOrder) -- not shedding, nothing runs, SPARE the sole compile within the allowance, not due, kill the youngest. While exactly one compile runs (the machine's held slots, which the watcher now hands the governor) and it holds no more than the allowance -- which the ceiling already bounds it to -- the shed kills nothing; two or more, or one past the allowance, and the youngest is killed as before. An unknown count spares nothing. The watcher says the spare once at Info as the verdict turns to it, never per dwell. Neutering the spare, its allowance bound, its sole-compile bound and the transition guard each turns red exactly the cases that guard it. Refs e392c45d Signed-off-by: Christian Parpart <christian@parpart.family>
…ery place that said "nothing new" e392c45d changed the rule from "a pressed machine admits nothing new, its own local compiles included" to "nothing beside a running compile, no other machine's work, and idle only its own, one at a time"; its review found the old sentence still in operator-visible strings and the docs. Every such place now says the new rule, and the shed's as well -- the youngest is killed but never a sole compile within the allowance: - metric help (MetricsCatalog yield/shed seconds, the pressure gauge in PrometheusFormatter) and their docs/tools/pace-node.md rows, word for word; the getting-started metrics row; - the fleet page's memory-pressure help (FleetView) and the Machines row; - the memory-pressure-shedding condition's remedy (NodeConditions), kept inside its wire ceiling; - the governor's layer-C table and bullets in pace-node.md, with two new ones: the spared sole compile, and the leader giving back what a settled compile was counted as; - comments: CompileCacheWire's PressureLevel, IMetricsSink, NodePolicy's MemoryPressure, FleetView's join and slot notes, MachineBudget. The rulebook (distributed-compilation.md, the compile-queue section) gains the two rules this work made: a pressed machine is never starved of its own compiles (with the three shapes that reopen it), and a stall confined to another cgroup's own memory limit is not the machine's. Signed-off-by: Christian Parpart <christian@parpart.family>
… some share PSI `full` for a cgroup is "every non-idle task of THIS cgroup stalled", so in a machine-wide freeze every busy cgroup reads about the system's figure -- a container at its own memory.max included. 1a1ef51f let a cgroup carrying 80% of the system's full alone be "confined": a genuine freeze beside such a container entered no shed (only a yield, then a shed after 20 s sustained), and a shed in force was left after one dwell while the freeze went on (re-review Important 1). `some` now decides ownership: the governor discounts a cgroup's `full` only where the same cgroup carries the `some` share, and the scan keeps and descends by `some` alone. The rulebook and pace-node.md say so, and state the cost the two-dwell evidence hold has: a shed after a cgroup leaves its limit is delayed by at most two dwells, not "never missed". Tests: a genuine freeze (confined full 8 of 8, some 20 of 40) sheds and discounts nothing; a shed in force with `some` below the yield threshold is not left on such a cgroup's `full`; the scan keeps no full-only carrier. Each was red before the fix; neutering the governor's some gate and the scan's some-only share each turns red exactly those cases. Signed-off-by: Christian Parpart <christian@parpart.family>
…ts it there The scan counted a cgroup within a twentieth of its memory.max as at its limit with no event rising -- the shape of page cache that filled up to the limit and stopped, which stalls on nothing of its own (re-review minor 1). Only a rise in the limit's memory.events(.local) count now says a cgroup stalls ON its limit. A rise is seen only between two looks, so a walk now keeps the carriers that SET a limit of their own (ConfinedStall::limited) and each sample re-judges them; memory.current is no longer read. Tests: a cgroup near its limit with no rise is never at it across samples; memory.high counts by its events alone; a first look is kept but not yet at its limit. Putting the `near` path back turns red exactly those three cases. Signed-off-by: Christian Parpart <christian@parpart.family>
…'s looks A walking sample looked at every candidate during the walk and then again microseconds later: the same files read twice, and the second look could see no event rise in between (re-review minor 3). Walk now returns what it kept, as it looked at them, and that is the sample's reading. The shared FakeCgroupHost counts reads per path; the new case asserts a candidate's memory.pressure is read once per sample, walking or not -- red before the change (two reads on the walking sample). Signed-off-by: Christian Parpart <christian@parpart.family>
…it no longer meets _eventsSeen kept one entry per (cgroup, limit) the scan ever looked at, for the life of the node -- and a machine starts and stops services all day (re-review minor 5). Each walk now keeps only the counts of the cgroups it looked at; every candidate is among them, so nothing a sample compares against is lost. Test: a service that disappears between two walks takes its two counts (max and high) with it -- red before the change (10 remembered, not 8). Signed-off-by: Christian Parpart <christian@parpart.family>
…hreshold The discount was named only while the system figure stood above the ENTRY threshold, so a stall hovering at the yield (or shed) threshold started and ended it, and logged both, every other sample (re-review minor 2). It now has the levels' own hysteresis: named from above the entry threshold and held, for the cgroup already named, down to the exit one. Test: a stall at 11, 9.5, 10.5, 9, 5.5 stays named; below the exit (4) it ends, and below the entry (9) a new one is not named yet -- red before the change (three assertions). Signed-off-by: Christian Parpart <christian@parpart.family>
christianparpart
force-pushed
the
feature/compile-oom-defense
branch
from
November 16, 2026 02:39
c35bc1f to
bd095a0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A machine running
pace-nodeand accepting fleet compiles while somebody uses it can run out of memory: the desktop freezes, then the kernel kills the wrong process. This change makes compiles the expendable part of the machine. They are capped, they yield memory when the machine needs it, and when memory runs out anyway a compile dies first. A killed compile is queued again where it fits instead of failing the build.The design is
docs/superpowers/specs/2026-11-13-compile-oom-defense-design.md. The three layers, in the order they were built:pace-nodegives its compiles a cgroup of their own (compiles/) beneath its delegated cgroup and caps them together at the machine's compile budget. Delegating the memory controller is on by default in both shipped units. The system unit runs in an emptypace.slicethat systemd-oomd does not pick from. That slice applies only after the service restarts.memory.max/memory.high, which counts only while that cgroup also carries the light-stall (some) share. A real machine-wide shortage therefore still yields and sheds, at most two dwells late.oom_score_adj=1000through a small shim, so it is the kernel's first OOM victim. The node itself leaves systemd-oomd's choice.Requeue. A compile killed for memory is answered with error
0x35(worker-compile-killed), carrying the peak it held. The client asks again withAskOriginAfterKill, and the leader queues it only on a machine whose budget fits that peak:AfterGrant/local-only. ARunsAlonegrant then runs it outside the ceiling.pace-ccfinds the local node's ceiling through a rendezvous file. It places a local compile under that ceiling when it fits, otherwise it compiles shim-only, and says which.Wire version 18 (master is 17):
0x35carries a peak trailer: the message,0x00, then a big-endian u64.AskOriginvalues: 0 first, 1 AfterGrant, 2 AfterKill.NeverFits=4 andRunsAlone=5.SettleOutcome::Killed=4.Observability:
IMetricsSink::Counteris nowuint16_t, because the counter enum outgrew a byte.Designer calls (C1–C9), as built:
memory.swap.max=0).memory.high.--compile-memory-limitlowers the reported budget.max(known, observed), and a shed victim's peak is not recorded. Refinement: the worker refuses the compile and forwards no peak for its own kill.Pre-existing bugs fixed along the way, one commit each, at the front:
FleetSampler::SampleOnceraced the sampler's own thread.Also in this PR: a settle now gives back what a mid-compile budget report counted, and a pressed machine no longer reserves for work it can never run. Without these, the "runs one at a time" floor managed one compile per 20 s report.
Closes #1770, closes #1771, closes #1772, closes #1773, closes #1774.
Testing. Unit tests cover each governor verdict, admission rule and wire value. The new rules were neutered one at a time, and each turned exactly its own cases red.
compile-oom-e2ekills a compile over its ceiling and checks that it is rebuilt locally.dist-compile-e2epasses on a host under real memory pressure. The local gate runs both legs alongside this PR's CI.