Skip to content

Compile OOM defense: cap, yield and shed compiles, kill them first, requeue what was killed - #1781

Merged
christianparpart merged 70 commits into
masterfrom
feature/compile-oom-defense
Nov 16, 2026
Merged

christianparpart merged 70 commits into
masterfrom
feature/compile-oom-defense

Conversation

@christianparpart

Copy link
Copy Markdown
Member

A machine running pace-node and accepting fleet compiles while somebody uses it can run out of memory: the desktop freezes, then the kernel kills the wrong process. This change makes compiles the expendable part of the machine. They are capped, they yield memory when the machine needs it, and when memory runs out anyway a compile dies first. A killed compile is queued again where it fits instead of failing the build.

The design is docs/superpowers/specs/2026-11-13-compile-oom-defense-design.md. The three layers, in the order they were built:

  • Layer B: one ceiling over all compiles. pace-node gives its compiles a cgroup of their own (compiles/) beneath its delegated cgroup and caps them together at the machine's compile budget. Delegating the memory controller is on by default in both shipped units. The system unit runs in an empty pace.slice that systemd-oomd does not pick from. That slice applies only after the service restarts.
  • Layer C: pressure shedding. A governor reads the kernel's memory-stall figures (PSI) and moves through three levels: calm, yielding (a lower allowance, nothing new beside a running compile) and shedding (the youngest compile is killed).
    • The compiles' own stall is discounted. So is a stall confined to one cgroup sitting at its own memory.max/memory.high, which counts only while that cgroup also carries the light-stall (some) share. A real machine-wide shortage therefore still yields and sheds, at most two dwells late.
    • Under shedding, a machine's sole compile is spared while it stays within the allowance.
    • An idle pressed machine still runs its own compiles, one at a time. Before this rule, a machine whose PSI never dropped (here, a host-wide stall from an unrelated process) held its whole build in the queue forever.
  • Layer A: kill priority. Every compiler is started with oom_score_adj=1000 through a small shim, so it is the kernel's first OOM victim. The node itself leaves systemd-oomd's choice.

Requeue. A compile killed for memory is answered with error 0x35 (worker-compile-killed), carrying the peak it held. The client asks again with AskOrigin AfterKill, and the leader queues it only on a machine whose budget fits that peak:

  • The peer-supplied peak is capped at the killing machine's budget.
  • After a second kill the client asks AfterGrant/local-only. A RunsAlone grant then runs it outside the ceiling.
  • pace-cc finds the local node's ceiling through a rendezvous file. It places a local compile under that ceiling when it fits, otherwise it compiles shim-only, and says which.

Wire version 18 (master is 17):

  • MACHINE-BUDGET has seven fields.
  • Error 0x35 carries a peak trailer: the message, 0x00, then a big-endian u64.
  • AskOrigin values: 0 first, 1 AfterGrant, 2 AfterKill.
  • New grant reasons: NeverFits=4 and RunsAlone=5.
  • New settle outcome: SettleOutcome::Killed=4.

Observability:

  • Conditions, counters and log lines name every cap, yield, shed and kill.
  • The fleet page shows each machine's pressure and what holds its budget.
  • Fleet totals leave out pressed machines.
  • The live-stats layout changed: IMetricsSink::Counter is now uint16_t, because the counter enum outgrew a byte.

Designer calls (C1–C9), as built:

  • C1. Compiles never swap (memory.swap.max=0).
  • C2. There is no static memory.high.
  • C3. A typed --compile-memory-limit lowers the reported budget.
  • C4. A SIGKILL-ended compile is a miss.
  • C5. A killed compile's peak is max(known, observed), and a shed victim's peak is not recorded. Refinement: the worker refuses the compile and forwards no peak for its own kill.
  • C6. The node is left out of oomd's choice.
  • C7. The ceiling excludes the node itself.
  • C8. The shed victim is the youngest compile.
  • C9. A yield reports at once.

Pre-existing bugs fixed along the way, one commit each, at the front:

  • FleetSampler::SampleOnce raced the sampler's own thread.
  • A toolchain walk cut short by a stop was cached.
  • E2E worker nodes shared a state directory.
  • Toolchain fingerprint cache entries written by older builds never retired.
  • The protocol docs said LEASE had three fields; it has six.

Also in this PR: a settle now gives back what a mid-compile budget report counted, and a pressed machine no longer reserves for work it can never run. Without these, the "runs one at a time" floor managed one compile per 20 s report.

Closes #1770, closes #1771, closes #1772, closes #1773, closes #1774.

Testing. Unit tests cover each governor verdict, admission rule and wire value. The new rules were neutered one at a time, and each turned exactly its own cases red. compile-oom-e2e kills a compile over its ceiling and checks that it is rebuilt locally. dist-compile-e2e passes on a host under real memory pressure. The local gate runs both legs alongside this PR's CI.

@christianparpart christianparpart added the type/feature New capability, or a user-visible extension of one label Nov 15, 2026
@github-actions github-actions Bot added area/launcher apps/fastcache-cc/ - the compiler launcher and its cache key area/compile-cache CompileCache/, the 0xFC wire, the executor, PathCanon area/distributed Distributed/, apps/fastcache-compile-node/ - dispatch, workers, scheduler area/protocol Protocol/, Auth/ - memcached, RESP, autodetect, the auth gate area/platform Platform/, Config/, Cli/ - service registration, config lookup, the CLI table area/observability Metrics/, logging, stats and scrape surfaces area/packaging packaging/, CPack, deb/rpm/pkg/msi, installers area/build CMake, presets, toolchains, CI workflows os/windows Specific to Windows (MSVC, clang-cl, Winsock, IOCP) os/linux Specific to Linux (glibc, epoll, systemd, deb/rpm) labels Nov 15, 2026
@christianparpart
christianparpart force-pushed the feature/compile-oom-defense branch 6 times, most recently from 54f28b8 to c35bc1f Compare November 15, 2026 23:27
A leader re-sends every entry a follower has not acknowledged on every
heartbeat (50 ms). The follower's in-memory log already skipped the ones
it held, but `RaftNode::OnAppendEntries` persisted from `prevLogIndex +
1` on any non-empty request, so each re-send truncated the durable log
and rewrote it: two flushes per heartbeat. Once a flush outlasted half
the interval the follower fell behind the re-sends for good -- its
reactor stayed in `SaveLog`, its acknowledgement never reached the
leader, and a joining voter was never promoted. It also cut acknowledged
entries on disk for the length of a rewrite, the hazard `TryAppend`
documents and avoids in memory.

`RaftLog::AppendOutcome` now says where the log CHANGED (`changedFrom`,
also the conflict's truncation point, nullopt for a run already held or
a rejected append), and the follower persists and re-derives its
configuration from there only.

Measured: "A two-voter fleet's follower holds its leader as a voter ..."
was red on Windows CI three times (2 members, 1 voter, stalled ~9.5 s),
and none of 2,368 local runs under CPU overcommit reproduced it (two
unrelated port collisions). With fsync slowed by an LD_PRELOAD shim
(gcc-release): 40/40 failed at 20-150 ms and 20/20 at 20-60 ms, with the
CI's exact message; 10-40 ms passed 20/20. After: 200/200 at 20-150 ms,
200/200 at 20-60 ms, 40/40 at 100-300 ms, and on clang-debug 200/200
(shim) plus 240/240 (plain). With the old persistence call restored the
three new RaftNode cases fail, and only they; the conflict at
prevLogIndex + 1 stays green as the control.

What remains is #1782: a follower still persists synchronously on the
reactor that reads its leader's session, so a long run of genuinely new
entries on a disk whose flushes outlast the heartbeat can still keep
that reactor from sending the acknowledgement.

Signed-off-by: Christian Parpart <christian@parpart.family>
…t that includes it

The clang-tidy job's "Analyse the changed lines" step hands every changed
header to clang-tidy-diff.py, which runs `clang-tidy -p=<db> <header>`. A
header has no compile_commands.json entry, so clang-tidy interpolates a
command from the entry whose PATH looks most like the header's, which knows
nothing about includes. For a test-only header that guess is often a unit
built without the include directories the header needs, and the run fails
`[clang-diagnostic-error] ... file not found` on a header that compiles in
every unit that includes it.

CI run 43735465655 (job 132877030688) failed that way on four headers:
src/tests/CgroupFakes.hpp ('Pace/Platform/CgroupV2.hpp' not found),
src/apps/pace-cc/OomVictimTestSupport.hpp, src/apps/pace-node/
AdmissionTestUtils.hpp and src/apps/pace-node/WorkerTierTestFixture.hpp
(catch2 not found). CgroupFakes.hpp is guessed wrong from master's own files
alone, so this predates the branch that found it. Reproduced locally with
clang-tidy 22.1.8 against the gate's clang-debug database.

scripts/tidy-header-commands.py writes a NEW database under $RUNNER_TEMP
(tidy-db), never the build tree's, so ninja's generated-header pass and
anything else reading the build tree see only what configure wrote. Every
input entry is copied unchanged; each selected header gets one entry
borrowed from an includer: the includer's directory and argument vector,
with the unit's source, `-c` and the `-o <object>` pair removed and
`-x c++-header <header>` appended (c++-header, so a header as a main file
draws no `#pragma once in main file`). The step points clang-tidy-diff.py's
-path at that database and keeps the `Failed:` grep and PIPESTATUS check.

The includer is found by RESOLVING each `#include` against the including
file's directory and every -I/-isystem/-iquote/-idirafter directory the
database names, following only files under src/. This deviates from the
suffix matching tidy-sweep.sh's BuildIncludeGraph uses: resolution needs no
listing of the tree (so no third-party-roots enumerator), and the files are
exactly the units the database names and what they reach. A breadth-first
walk over the reverse graph takes the first depth holding a database unit,
so a direct includer beats a transitive one; ties prefer the unit whose stem
is the header's (Foo.hpp -> Foo.cpp / Foo_test.cpp), then the smallest path.
A header no unit includes gets no entry, so clang-tidy still interpolates
its command -- but over the AUGMENTED database, whose borrowed header entries
are candidates too, so the guess is not necessarily the one it made before.
Such a header is now NAMED in the step's output; it is never dropped from
the analysis. The header extensions are written once in the step: one case
arm both selects a header and collects it for the script.

Limits. GNU-style driver lines only: the Linux leg calls this,
clang-tidy-windows does not, and the self-test is registered `if(NOT WIN32)`.
The include directories searched are the UNION over every unit, so a bare
quoted spelling can resolve through another target's directory and name a
false includer whose command lacks the header's own -I. Paths are compared
after abspath/normpath, never realpath, so a work tree reached through a
symlink fails safe: every header is named as having no includer and keeps
the interpolated command.

Measured: the four headers get the commands of CgroupV2_test.cpp,
OomVictimExec_test.cpp, AdmissionResponder_test.cpp and
NodeConditions_test.cpp. clang-tidy-diff.py, run as the step runs it on
`git diff -U0 origin/master...HEAD` of the four, exits 1 with the
clang-diagnostic-error on the build tree's database and 0 with no finding on
the augmented one. A probe database adding `-include/nonexistent-probe.h` to
each borrowed entry made clang-tidy fail on that include for all four, so the
borrowed entries are the ones used rather than a guess that happens to work.
Over all 40 headers the branch changes, all 40 found an includer and
clang-tidy-diff.py reports nothing.

The tidy-header-commands-selftest ctest (hygiene, default set, not on
Windows) drives a synthetic tree and database built with `arguments`, so a
scratch path containing a backslash passes 10/10: a direct includer's
command with the source swapped, the same-stem unit preferred although it
is the LARGER path, a direct includer (Z_user.cpp) beating a deeper unit
that would win the tie-break, a header reachable only transitively, a
header with no includer getting no entry and being named, the existing
units unchanged. Neutering each failed exactly the cases covering it:
the stem key replaced by `key=lambda unit: unit` (1), collecting every
reachable unit before the tie-break (1), the source swap (1), the naming
(1), and the transitive walk (3).

Signed-off-by: Christian Parpart <christian@parpart.family>
…ds the tests

gcc 16.2 at -O3 refuses McpNodeTools_test.cpp under -Wnull-dereference:
`*values.front().AsString()` dereferences a pointer the optimiser cannot
prove non-null, the same shape 5e17981 converted elsewhere. Two more
files carry it unguarded (McpToolchainTools_test.cpp, McpActionTools_test.cpp);
they are converted with it, so the next gcc does not find them one at a time.

Signed-off-by: Christian Parpart <christian@parpart.family>
…thread

SampleOnce documents itself as "take one sample now, whatever the timer is
doing", and nineteen test calls drive it while the thread the constructor
starts samples too. A sample reads its sources and only then records, and a
history keeps its newest record as the reading the next delta is taken from.
Two unserialised samples could therefore record out of the order they read in.

Measured with traces: the thread read DispatchGranted=0 before the case's
queued grant and recorded it after the case's own sample had recorded 1.
That overwrote the t1 bucket's raw value and `_last` with the stale 0, so
"A compile the admission queue placed on a worker reads as dispatched on the
fleet page" read "dispatched":{"value":0. Production has one caller (the
thread), so no shipped behaviour changes; the seam's promise was false.

The fix holds one mutex across a whole sample: every read, every record and
the keyed histories' fold.

New case "Two samples taken at once never overlap so a stale reading cannot
be recorded last" asks the property directly. The injected node-facts
provider, called inside a sample before it records, counts samples inside
at once. The first entrant waits (bounded, 500 ms) for a second, while the
case and a helper thread both sample; the peak is asserted after the join.
Lock removed: ran=100 fail=100 idle and ran=100 fail=100 under load.
Lock present: ran=100 fail=0 idle and ran=100 fail=0 under load.

Rates for the dispatched case (scripts/flake-rate.sh, clang-debug, 32 busy
loops on 32 cores):
  before: ran=150 pass=143 fail=7  (idle host: ran=50 fail=0)
  after:  ran=150 pass=150 fail=0  (idle host: ran=100 fail=0)

Signed-off-by: Christian Parpart <christian@parpart.family>
A node told to stop while its toolchain walk was hashing skipped the rest
of the files in AnnouncingParallelFor and still answered IParallelFor::Run
with "every slice ran". ProbeToolchainFiles therefore reported the scan
complete, and CachedToolchainFingerprint wrote the short digest into the
machine-wide fingerprint cache under a stamp that validates forever -- the
stamp folds each root's path and mtime, never its contents.

Every later node on that machine then served a fingerprint no client
computes. dist-compile-e2e reads its worker's first `serving ... as` line
while isolating only its launcher's state, so it failed on the mismatch
and, by tearing its worker down mid-walk, poisoned the entry again for the
next run. The code is unchanged from master; master fails the same way
against the same cache state.

A skipped slice now makes Run answer false, so the walk is incomplete and
nothing is cached, as IParallelFor documents for a slice that did not
finish. The new case stops the survey from inside the include probe, so
the stop lands in a walk that has begun -- the existing stop case uses a
token stopped beforehand and never reached the walk.

Signed-off-by: Christian Parpart <christian@parpart.family>
…s own

dist-compile-e2e isolated its launcher's state but not its nodes'. A worker
fingerprints its toolchains through the launcher's code into the launcher's
machine-wide cache, so the fixture's worker served whatever the developer's
~/.local/state/pace-cc/toolchains held and wrote into it: one truncated
entry there failed the fixture run after run and read as a branch
regression (#1774).

The node now gets the launcher's seam. e2e_node_state_enter writes a shim
through the same writer and the same StateDirectoryImpl rows as
e2e_launcher_state_enter, over <workdir>/node-state, and points the node
variable at it; New-E2ENodeState / Use-E2ENodeState do the same around one
child in PowerShell. The root is deliberately NOT the launcher's: with one
root the launcher would read the worker's entry instead of walking, and the
fixture's launcher/worker agreement check would compare one answer with
itself. The positive control is the worker's walk caching HERE
(e2e_node_state_assert_used --fingerprinted, Get-E2ENodeFingerprintCount).

Every fixture whose node runs a worker, and so a toolchain survey, is
converted: dist-compile-e2e.sh, fleet-dashboard-e2e.sh,
node-socket-activation-e2e.sh, dist-compile-e2e.ps1 and
node-scratch-isolation-e2e.ps1. node-mcp-e2e.sh already isolated its
workers by hand; the rest start only --slots=0 nodes or nodes refused at
startup, which survey nothing.
rpm-package-e2e.sh and macos-package-e2e.sh are out of scope: they install
the package and run the node as the packaged service, inside a throwaway
container or under the system service account, so the state directory it
caches into is that container's or that account's, never the developer's.

check-node-fixture-starts now enforces the seam: a `runs` start naming no
--slots=0 and no inline XDG_STATE_HOME= is reported `unisolated-node
<file>:<line>` unless a .sh entered e2e_node_state_enter above it or a .ps1
start sits inside Use-E2ENodeState. exit-codes-e2e and node-config-file-e2e
are exempt by row: their starts are refused by the startup rules before any
tier is built. e2e-helpers-selftest drives the control's pass, foreign-root,
zero-run, empty-record and not-walked verdicts through the real shim.

The shim redirects HOME as well (LOCALAPPDATA on Windows) for the node and
every compiler it runs, which reaches further than the fingerprint cache;
the library and the rule say so.

Refs #1774

Signed-off-by: Christian Parpart <christian@parpart.family>
…wrote

297a4e0f stopped a node from caching the digest of a walk a stop cut
short, but it reached only NEW entries. One already written keeps
validating, because the stamp folds each root's path and mtime and never
its contents, and nothing in an entry tells a truncated digest from a whole
one -- so every node and launcher on such a machine kept serving a
fingerprint no client computes until something forced a full walk.

The stamp's own schema moves from toolchain-stamp-v2 to v3, so every entry
an older build wrote is a miss: rewalked once and written back. The
fingerprint VALUE is untouched, so a fleet mid-upgrade still matches; the
cost is one walk per toolchain per machine. The schema is now a public
ToolchainStampSchema and a defaulted parameter of ComputeToolchainStamp, as
FingerprintSchema already was, so a test can stamp an entry the way the
older build did.

The new case plants a wrong fingerprint under a v2 stamp beside a control
planted under v3, which is served. With the constant put back to v2 (and
its guard REQUIRE removed) the case's four behavioural checks fail and
nothing else in [toolchain-probe] does.

Closes #1774

Signed-off-by: Christian Parpart <christian@parpart.family>
The verb table showed LEASE as [fingerprint][objectKey][codecs], which was
already stale before this branch: EncodeLease sends six fields -- fingerprint,
objectKey, codecs, exclusions, toolchainLabel, origin. The table now shows all
six and a paragraph says how each is encoded: codecs one byte per id, the
exclusions a nested list of at most 16 endpoints (MaxLeaseExclusions) of 1 to
261 bytes, refused rather than truncated past that, the label UTF-8, and the
origin one byte below AskOrigin::Last.

Signed-off-by: Christian Parpart <christian@parpart.family>
Signed-off-by: Christian Parpart <christian@parpart.family>
Signed-off-by: Christian Parpart <christian@parpart.family>
A compiler the worker's machine killed -- the compile memory ceiling, the
node's own shedding, or any other SIGKILL -- answered nothing about the
code, yet reached the client as a CompileOutcome with exit 137: a compiler
failure in a TU that compiles. It is now JobRefusal::CompileKilled, answered
worker-compile-killed (0x35, a new byte and no version bump: the reply
grammar is unchanged and an older launcher reads an unknown refusal and
compiles locally all the same), graded WorkerRefused by the launcher.

One CompileKillTable names each cause (ceiling, shed, killed) and its
counter: pace_worker_compiles_killed_{ceiling,shed,signal}_total. The node
logs one Info line per kill naming the cause and the source
(KilledCompilesLog, a decorator in front of RefusedArgumentsReport).

The worker reports NO peak for a killed compile: it was cut short, so its
peak is a lower bound, and the client's local recompile settles the
estimate with the true figure. This refines the spec's call C5
(max(known, observed)) and must be called out in the PR.

The live-stats layout pin moves with the three counters.

Signed-off-by: Christian Parpart <christian@parpart.family>
…able

Signed-off-by: Christian Parpart <christian@parpart.family>
The worker now writes compiles/memory.max at startup -- unconditionally, so
a compiles/ an earlier run left behind does not keep that run's figure --
and again after every accepted reload. The ceiling is the new
--compile-memory-limit=<size|N%> (YAML compile_memory_limit, reloadable,
local wiring), else the machine budget MachineBudgetOf reports, so
admission and the cap are one number (CompileCeilingOf). Typed below that
budget it lowers the reported budget too (C3).

A worker whose ceiling cannot be held raises the new latched
compile-memory-uncapped condition beside compile-memory-unmeasured, with
the preparation's (or the write's) failure as its detail; it is answered
only by PutCompileCeilingInForce, so a wiring that forgets the ceiling
reads undecided. The condition table spent its spare row: MaxNodeConditions
26, bought with one more KiB of HistoryPayloadShare, no field ceiling moved.

PrepareJobCgroups no longer swallows every refusal on compiles/'s
memory.swap.max and memory.oom.group: only an absent or unsupported file is
tolerated, listed on CgroupLayout::notApplied and said in one Warn line at
startup; anything else is CompilesNotCreated. A job's memory.oom.group
refused for another reason fails Create and removes the directory.

Signed-off-by: Christian Parpart <christian@parpart.family>
…y of its own

compile-oom-e2e.sh started a worker with the caller's state directory, so its toolchain walk
wrote the developer's fingerprint cache, and `node-fixture-starts` refused the tree: it is the
one fixture #1774's isolation seam did not reach. Entered after the launcher's, as every other
fixture that starts a worker does.

Signed-off-by: Christian Parpart <christian@parpart.family>
…t fits

A compile asked for again after its compiler was killed for memory says so on its ADMIT:
`AskOrigin::AfterKill`, a third value of the byte ADMIT and LEASE already carry. The kill itself
reaches the leader the way it already did -- the launcher ends the admission unsettled, and
`Abandon` raises the key's estimate (`RecordKilled`) -- so the requeue is charged that.

The queue then places such a request only on a machine whose CONFIGURED budget holds its
estimate. A smaller one is no target at all, where a first request would run alone on it: there
the compile ceiling is the budget, and the run would be the ceiling's next kill. A budget the
pressure allowance holds down is waited for, never ruled out. When every reported machine the
request could run on is too small, its caller included, it is decided `neverFits` and granted
`Local` with the new `LocalGrantReason::NeverFits`, alone on its caller -- the client runs it
outside the ceiling.

Both enumerators land in wire version 18, which is not yet released: the precedent of its
MACHINE-BUDGET fields, and a version-17 decoder refuses either as past its `Last`. Counted by
two catalogue rows the admission surface writes, `pace_admission_requeued_after_kill_total` and
`pace_admission_never_fits_total`; the live-stats layout pin and the attribution figures move.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…n it

The seam for the compiles `pace-cc` runs itself. A node names its `compiles/` cgroup in a
rendezvous file in its runtime directory (`RendezvousFile`, written atomically, quiet on
repetition); a launcher reads `$XDG_RUNTIME_DIR/pace-node/compile-ceiling` (`RendezvousLocator`),
which is that file exactly under the user unit, the one place the two share an owner. A node with
no `RUNTIME_DIRECTORY` names nothing, so a node started by hand can never overwrite or withdraw
the rendezvous of the node the user runs.

`LocalCompileRunner` places a compile in a cgroup of its own below that ceiling
(`CgroupJobSource`, now with a per-owner name prefix so a launcher's `pace-cc-<pid>-<n>` never
takes a node's `job-<n>` and the node's startup sweep never takes a launcher's), and runs it under
the OOM-victim shim wherever there is one. Which applied is a named outcome per compile --
`LocalDefense` and, when not the ceiling, the `CeilingMiss` that says why -- read off whether the
child actually started in the cgroup (`CompileRun::inCgroup`, set by the runner) and off what the
cgroup source heard, never assumed.

`CgroupV2.cpp` (std-only) is compiled into the launcher for it, and so is the shim, which the
launcher now spawns through.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…s under the ceiling

A worker's `worker-compile-killed` no longer compiles here at once, outside every layer of the
defense. It is its own decline cause (`MemoryKilled`, `IsWorkerCompileKilled`), and the miss path
ends that admission UNSETTLED -- which is how the leader learns a kill -- and asks once more as
`AskOrigin::AfterKill`, so the queue places it where its estimate fits, now or once room frees.

One requeue per compile, whoever killed it (`KillStep`, `RunMiss`):
- a second worker kill compiles here, asked for local-only with that worker excluded, tallied
  `killed for memory twice, so compiled locally`;
- a `NeverFits` grant runs here OUTSIDE the ceiling, tallied apart;
- a compile the local ceiling kills waits in the queue once in the same way, and killed there
  again runs once more outside the ceiling under the same grant;
- a kill outside every ceiling ends the compile killed, said on stderr whatever the verbosity,
  and teaches the leader nothing of a lower-bound peak.
With no queue to wait in, a compile the ceiling killed runs once more outside it at once.

Every compile the launcher runs itself goes through `LocalCompileRunner`, so it joins the local
node's ceiling where it can and starts as the first OOM victim everywhere else -- the launcher is
its own `pace-oom-victim-exec` shim -- and the invocation log records which in a new
`local-defense` column (format v4; a v3 line is read as its prefix), which `-s` counts.

The real compile-OOM fixture asserts the requeue: on its one 256m machine the kill is followed by
`asking the queue once more`, and then never-fits or a second kill.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…self

While its ceiling holds, a node writes its `compiles/` path to `compile-ceiling` in its runtime
directory, and withdraws it when the ceiling does not hold -- from `PutCompileCeilingInForce`,
the one place a ceiling is put in force, through the keeper every later write goes through. The
rendezvous rides the process's one `CompileMemoryMeter`, so a formation reform keeps it. A
failure to write it is said at Info once; the ceiling works without it.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…de's ceiling

`pace-node.md` gains the two sections the behaviour needs -- a killed compile waits for memory,
and the compiles pace-cc runs itself -- and its kill sentences stop saying the client compiles
locally at once. `pace-cc.md` names the three new tally rows, the `local-defense` column and its
format version, and the two queue rules. The rulebook states both as rules of the compile queue
and drops #1770 from its open work.

Closes #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
A runner given a cgroup source answered no peak at all for a child that ran outside its cgroup --
no cgroup could be made, or the system would not start the child in one. That is right for a node,
whose `ru_maxrss` includes its own resident set, and wrong for the launcher since #1770 gave it a
cgroup source: every local compile in an ssh session's scope, or under a node whose `compiles/`
refused this user, settled with no peak, and the estimator stopped learning what it learned
before the ceiling existed.

The fallback is now the runner's own peak source (`MakeProcessRunner(cgroups, shim,
outsideCgroup)`): `Unmeasured` by default and for a node, `ResourceUsage` for the launcher.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…hold

After the second worker kill the miss path asks local-only, that worker excluded -- and asked it as
`AfterKill`. For a local-only request the leader judges never-fits against the caller ALONE, so a
6 GiB laptop whose 32 GiB workers had both held the compile was answered `NeverFits`: the leader
counted `pace_admission_never_fits_total`, and the client's grant seam overwrote its
`killed for memory twice` tally with *no machine's budget fits it*.

The local-only fallback after a failed or killed worker now asks `AfterGrant` (`Waits::LocalOrigin`):
the same compile asked again, which is what it is. `AfterKill` stays the one requeue's, judged against
every machine the request could run on.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
The requeue after a kill was charged `max(known, default)` -- for a key never measured, exactly what
its first ask was charged -- so a compile that needed more was placed on an equal machine, killed
again and compiled locally: "wait until space is free" was "try once more".

The kill now tells the leader what the compile held when it died:
- a worker's `worker-compile-killed` reply carries it as an error TRAILER -- eight big-endian bytes
  after a `0x00` that ends the message (`EncodeKilledPeakTrailer`, `ErrorTrailerOf`,
  `KilledPeakOf`), so every reader of the message reads exactly what it read before;
- the client settles the admission `SettleOutcome::Killed` with it, and with its own cgroup's peak
  for a compile killed here, instead of ending it unsettled;
- the leader folds it in as a LOWER bound (`PeakEstimates::RecordKilled`: raise, never replace,
  never a measurement), so the requeue is charged at least what it held and placed only where a
  configured budget holds that.

The trailer and the outcome land in unreleased wire version 18, as the rest of #1770 does.
`wire-tag-reach-count` moves to 312 for the trailer's case.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…he ceiling waits once too

`pace-node.md` said a compile killed outside every ceiling is not retried; its FIRST kill is
requeued once like any other, and only a second one ends the compile. The kill sections now say the
admission is settled `Killed` with what the compile held rather than ended unsettled, that the
fallback after a second kill is never counted never-fits, and that a compile outside the ceiling is
still measured. The rulebook bullet says the same.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…it at once

Since the fallback after a second kill asks `AfterGrant`, its grant came back `Local` with reason
`Asked` and the launcher joined this machine's ceiling: with the estimate the kills taught above the
machine's budget -- which IS its ceiling -- that run was killed a third time before `RunHeld` tried
outside. A wasted compile, and the machine driven to its ceiling, on every killed-twice TU.

The leader now says why such a grant runs alone: `LocalGrantReason::RunsAlone` (v18, unreleased),
for a `Local` decision that ran alone and is no never-fits -- counted as nothing beyond the
`AdmissionRanAlone` it already is, so never-fits stays at zero. The launcher's `LocalGrantTable`
gains two placement columns, before and after a kill (`PlacementForGrant`): `RunsAlone` keeps the
ceiling for a first ask, whose estimate is a guess, and runs outside it once a kill taught the
estimate; `NeverFits` runs outside always.

Also: the comments that still said a kill ends the admission unsettled, and `EncodeErrorReply`'s
Doxygen block, which sat above `ErrorTrailerSeparator`.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
What a killed compile held is a figure a peer states, and `RecordKilled` raises the key's estimate
and its source shape's maximum for good: one forged trailer could make every configuration of that
source run alone. A compile cannot hold more than the machine it ran on lets its compiles hold --
its ceiling is that budget -- so `AdmissionQueue::HeldBound` clamps the figure to the budget that
machine last reported. The charged-at-least case is reworked to a kill within that bound.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…ys otherwise

`--compile-memory-limit`'s help, the reference configuration's two mirrors of it and of the yield,
the worker's kill log line and the yield row of `pace-node.md` still said a killed compile's client
compiles it locally. It asks the queue for it once more, charged what it held. `pace-node.md` also
says when a killed-twice compile runs outside its machine's ceiling, and what bounds a kill's figure.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
… request

Comment and reference-configuration text only, no behaviour change: `memory_pressure_shed` said a
shed compile's client compiles it locally, where it asks the queue once more; `AdmitGrant::reason`
said only a dispatchable request is given a reason, where `NeverFits` and `RunsAlone` answer
`LocalOnly` too; `LocalPlacement::OutsideCeiling` did not name a `RunsAlone` grant after a kill;
a stray blank line sat between `EncodeErrorReply`'s Doxygen and its declaration; and the
`compile_memory_limit` comment ran past the file's width.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
… v18 ADMIT/SETTLE values

The `0x35` worker-compile-killed row still said the client compiles locally and
the worker reports no peak. It now names the trailer (0x00, then the peak as an
8-byte big-endian u64), the SETTLE `Killed` outcome carrying it, and the one
`AfterKill` requeue, with local fallback only for never-fits or a second kill,
all inside wire version 18. ADMIT gains its `origin` byte, the grant reasons
`4` never-fits and `5` runs-alone, and SETTLE outcome `4` killed.

Refs #1770

Signed-off-by: Christian Parpart <christian@parpart.family>
…, one at a time

dist-compile-e2e hung in case 9 on a host whose memory PSI was high for a
reason no compile caused: another cgroup thrashing at its own memory.max.
The cache-only node (--slots=0), leader of its own one-machine fleet,
reported Shed in its MACHINE-BUDGET, and its launcher's ADMIT waited for as
long as the pressure lasted -- here, forever (`admit -> no-reply`).

Introduced by 2846351d (feat(distributed): admission holds back a machine
under memory pressure), which made AdmissionQueue::Pump's grant predicate
answer `false` for every target reporting a non-Calm level, the caller's
own idle machine included; e984ab4e widened it from Shed to Yield, and
eea25fba made it reachable, since from then every node reports the level
the host's PSI implies, even with no compiles cgroup to act on. The queue
promises a miss is never starved (#1728); on an idle machine waiting
relieves nothing the fleet holds, and nothing it does can end it.

A pressed machine now takes nothing new while any compile of the fleet
runs or is reserved there, as before, and no other machine's work at all;
idle, it runs its OWN next compile that fits the budget it reports -- the
allowance, which 2d24e1ba already floors at one default estimate so that
exactly one compile fits. A compile larger than that still waits, as the
closed run-alone exception requires. The shared idle conjunct is one
`idle` for both that floor and run-alone.

Tests: AdmissionQueue gains the floor (Yield and Shed, one at a time), its
scope (no other machine's work, nothing oversize), and the shed case now
holds a ticket beside a RUNNING compile; AdmissionResponder pins both at
the wire. Neutering each conjunct of the floor turns red exactly the cases
that guard it. The docs, the wire header and the governor's log line say
"beside a running compile".

Signed-off-by: Christian Parpart <christian@parpart.family>
… machine's

The governor read root PSI as machine pressure, and a service stalling at
its own memory.max raises that as surely as a shortage does -- with GiBs
free. On the host dist-compile-e2e ran on, one user service doing exactly
that held every node at Shed (system full 14.8%), which freeing memory
cannot relieve and which yielding compiles cannot end.

The source now finds such cgroups (Platform::ConfinedStallScan, through
the ICgroupHost seam): a cgroup outside this node's own subtree, its
ancestors and the compiles', that carries 80% of the machine's some or
full stall while standing at its own memory.max or memory.high (an
OwnLimitTable row each) -- memory.events count risen since the previous
sample, or memory.current within a twentieth of the limit. The governor
counts such a stall the way it counts the compiles' own: it neither
yields nor sheds and eases a level, with at-limit evidence held two
dwells per cgroup. One cgroup must carry the share alone, so a real
shortage, which stalls the rest of the machine too, still sheds.

Cost and staleness: the tree is walked only while the machine stalls at
all, at most once per 10 s, descending only into a cgroup carrying the
share (PSI covers the subtree), capped at 64 cgroups and 8 levels; each
sample re-reads at most four kept candidates. A cgroup reaching its
limit after a walk counts as the machine's until the next one -- a yield
it did not need, never a freeze missed. Control files are told from
child cgroups by the directory's own cgroup.controllers.

An unlistable tree is no confined stall at all and is said once at Warn
(and once at Info when readable again); the discounted cgroup is named
at Info when the discount starts or ends, never per sample. Neutering the
shed-entry discount, the yield discount, the evidence lapse, the own
exclusion, the at-limit requirement, the pruning, the walk interval, the
calm guard and both watcher transitions each turns red exactly the cases
guarding it.

Signed-off-by: Christian Parpart <christian@parpart.family>
memory.events is hierarchical: a child reaching its own limit counts in
every ancestor's file too, so a slice with a numeric limit far above
what it holds read as "at its limit" whenever a child of it was. The
confined-stall scan now reads memory.events.local where the kernel
offers it (5.8 and later), the hierarchical file only where it does not.
Neutering the order turns red the one case that guards it.

Signed-off-by: Christian Parpart <christian@parpart.family>
…it, and a settle frees what a report counted

Two ways e392c45d's floor -- an idle pressed machine still runs its own
compiles, one at a time -- was still starved, from its review:

- Another machine's dispatchable request whose candidates named a pressed
  worker became that worker's head and, a pump later, reserved its whole
  budget there. The hold kept the worker from ever being idle, so its own
  launcher waited as long as the other caller stayed blocked: case 9's
  hang, moved to a fleet. A request that can never run on a pressed
  machine -- another machine's work, or one above the budget it reports --
  is now neither its head nor reserves there (BarredWhilePressed, which
  the grant check reads too, so the rule has one definition).
- The ledger counts max(charged, reported held). A report that arrived
  while a granted compile ran counted it, and after the settle kept the
  machine non-idle until the next report: the floor ran one compile per
  20 s. A settle now gives back what a report received AFTER its grant
  counted (Charge::reportsAtGrant); one taken before the grant owes
  nothing. A report taken before a grant but arriving after it would lose
  another compile's count instead -- fail-open for one report period at
  most, against fail-closed for every one.

Tests: the floor's second request is now held by `idle` alone (1 GiB on
a 2 GiB budget); a machine reporting no slot runs nothing; the shed case
pins charged, report-only held slots and report-only held bytes apart;
the reservation and give-back cases with a control for the report order.
Neutering the skip, the slot conjunct, each used-* conjunct, the order
condition and the slot give-back each turns red exactly its case. The
hold.* half of `idle` stays green under neutering: after this change a
pressed machine's holds come only from its own fitting head, which is
blocked only while the machine is not idle anyway.

Refs e392c45d

Signed-off-by: Christian Parpart <christian@parpart.family>
Under Shed the governor killed the youngest compile once per dwell
whenever compiles held anything. With e392c45d's floor that is the one
own compile a pressed machine still runs: killed, requeued AfterKill,
admitted again, killed again, and run OUTSIDE every ceiling after the
second kill (MissFlow) -- every own TU compiled three times during a
real freeze, ending with no memory bound at all.

The owner's ruling, as a table: what a shed does about the running
compiles is a ShedKill verdict (ShedKillTable, RowsInEnumeratorOrder) --
not shedding, nothing runs, SPARE the sole compile within the allowance,
not due, kill the youngest. While exactly one compile runs (the machine's
held slots, which the watcher now hands the governor) and it holds no
more than the allowance -- which the ceiling already bounds it to -- the
shed kills nothing; two or more, or one past the allowance, and the
youngest is killed as before. An unknown count spares nothing. The
watcher says the spare once at Info as the verdict turns to it, never
per dwell.

Neutering the spare, its allowance bound, its sole-compile bound and the
transition guard each turns red exactly the cases that guard it.

Refs e392c45d

Signed-off-by: Christian Parpart <christian@parpart.family>
…ery place that said "nothing new"

e392c45d changed the rule from "a pressed machine admits nothing new, its
own local compiles included" to "nothing beside a running compile, no
other machine's work, and idle only its own, one at a time"; its review
found the old sentence still in operator-visible strings and the docs.
Every such place now says the new rule, and the shed's as well -- the
youngest is killed but never a sole compile within the allowance:

- metric help (MetricsCatalog yield/shed seconds, the pressure gauge in
  PrometheusFormatter) and their docs/tools/pace-node.md rows, word for
  word; the getting-started metrics row;
- the fleet page's memory-pressure help (FleetView) and the Machines row;
- the memory-pressure-shedding condition's remedy (NodeConditions), kept
  inside its wire ceiling;
- the governor's layer-C table and bullets in pace-node.md, with two new
  ones: the spared sole compile, and the leader giving back what a
  settled compile was counted as;
- comments: CompileCacheWire's PressureLevel, IMetricsSink, NodePolicy's
  MemoryPressure, FleetView's join and slot notes, MachineBudget.

The rulebook (distributed-compilation.md, the compile-queue section)
gains the two rules this work made: a pressed machine is never starved
of its own compiles (with the three shapes that reopen it), and a stall
confined to another cgroup's own memory limit is not the machine's.

Signed-off-by: Christian Parpart <christian@parpart.family>
… some share

PSI `full` for a cgroup is "every non-idle task of THIS cgroup stalled",
so in a machine-wide freeze every busy cgroup reads about the system's
figure -- a container at its own memory.max included. 1a1ef51f let a
cgroup carrying 80% of the system's full alone be "confined": a genuine
freeze beside such a container entered no shed (only a yield, then a
shed after 20 s sustained), and a shed in force was left after one dwell
while the freeze went on (re-review Important 1).

`some` now decides ownership: the governor discounts a cgroup's `full`
only where the same cgroup carries the `some` share, and the scan keeps
and descends by `some` alone. The rulebook and pace-node.md say so, and
state the cost the two-dwell evidence hold has: a shed after a cgroup
leaves its limit is delayed by at most two dwells, not "never missed".

Tests: a genuine freeze (confined full 8 of 8, some 20 of 40) sheds and
discounts nothing; a shed in force with `some` below the yield threshold
is not left on such a cgroup's `full`; the scan keeps no full-only
carrier. Each was red before the fix; neutering the governor's some gate
and the scan's some-only share each turns red exactly those cases.

Signed-off-by: Christian Parpart <christian@parpart.family>
…ts it there

The scan counted a cgroup within a twentieth of its memory.max as at its
limit with no event rising -- the shape of page cache that filled up to
the limit and stopped, which stalls on nothing of its own (re-review
minor 1). Only a rise in the limit's memory.events(.local) count now
says a cgroup stalls ON its limit.

A rise is seen only between two looks, so a walk now keeps the carriers
that SET a limit of their own (ConfinedStall::limited) and each sample
re-judges them; memory.current is no longer read. Tests: a cgroup near
its limit with no rise is never at it across samples; memory.high counts
by its events alone; a first look is kept but not yet at its limit.
Putting the `near` path back turns red exactly those three cases.

Signed-off-by: Christian Parpart <christian@parpart.family>
…'s looks

A walking sample looked at every candidate during the walk and then again
microseconds later: the same files read twice, and the second look could
see no event rise in between (re-review minor 3). Walk now returns what it
kept, as it looked at them, and that is the sample's reading.

The shared FakeCgroupHost counts reads per path; the new case asserts a
candidate's memory.pressure is read once per sample, walking or not --
red before the change (two reads on the walking sample).

Signed-off-by: Christian Parpart <christian@parpart.family>
…it no longer meets

_eventsSeen kept one entry per (cgroup, limit) the scan ever looked at,
for the life of the node -- and a machine starts and stops services all
day (re-review minor 5). Each walk now keeps only the counts of the
cgroups it looked at; every candidate is among them, so nothing a sample
compares against is lost.

Test: a service that disappears between two walks takes its two counts
(max and high) with it -- red before the change (10 remembered, not 8).

Signed-off-by: Christian Parpart <christian@parpart.family>
…hreshold

The discount was named only while the system figure stood above the
ENTRY threshold, so a stall hovering at the yield (or shed) threshold
started and ended it, and logged both, every other sample (re-review
minor 2). It now has the levels' own hysteresis: named from above the
entry threshold and held, for the cgroup already named, down to the exit
one.

Test: a stall at 11, 9.5, 10.5, 9, 5.5 stays named; below the exit (4) it
ends, and below the entry (9) a new one is not named yet -- red before
the change (three assertions).

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart
christianparpart force-pushed the feature/compile-oom-defense branch from c35bc1f to bd095a0 Compare November 16, 2026 02:39
@github-actions github-actions Bot added the area/cluster Consensus/, Cluster/ - Raft, discovery, membership label Nov 16, 2026
@christianparpart
christianparpart merged commit 464e258 into master Nov 16, 2026
33 checks passed
@christianparpart
christianparpart deleted the feature/compile-oom-defense branch November 16, 2026 04:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build CMake, presets, toolchains, CI workflows area/cluster Consensus/, Cluster/ - Raft, discovery, membership area/compile-cache CompileCache/, the 0xFC wire, the executor, PathCanon area/distributed Distributed/, apps/fastcache-compile-node/ - dispatch, workers, scheduler area/launcher apps/fastcache-cc/ - the compiler launcher and its cache key area/observability Metrics/, logging, stats and scrape surfaces area/packaging packaging/, CPack, deb/rpm/pkg/msi, installers area/platform Platform/, Config/, Cli/ - service registration, config lookup, the CLI table area/protocol Protocol/, Auth/ - memcached, RESP, autodetect, the auth gate os/linux Specific to Linux (glibc, epoll, systemd, deb/rpm) os/windows Specific to Windows (MSVC, clang-cl, Winsock, IOCP) type/feature New capability, or a user-visible extension of one

Projects

None yet

1 participant