Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .agent/rules/compile-cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,9 @@ same on both — the same defect with no MSVC anywhere near it.
platform. `toolchain-v2` and `toolchain-cache-name-v2` moved together. `toolchain-v3`
(#1656, which trees are folded) moved alone, because the stamp folds the schema
since then, so a bump misses every older cache entry without renaming the file.
`toolchain-v4` (the machine's `/usr/include` and `/usr/local/include` dropped)
moved alone for the same reason; no `objkey-v*` or `manifest-v*` moves with a
fingerprint bump, because no cache key folds the fingerprint.
- A test staging two different include trees would pass under the bug. The banner and
the file list are held EQUAL and the grammar is the only difference, with the
same-grammar case asserted beside it — a digest that separated everything would pass
Expand Down
21 changes: 19 additions & 2 deletions .agent/rules/distributed-compilation.md
Original file line number Diff line number Diff line change
Expand Up @@ -3159,8 +3159,19 @@ Which tree a fingerprint walks is `DriverSpec::fingerprintTree`, a column, never
on the driver: `cl` folds **none** (its banner names toolset and target, #195), a clang
whose `-v` probe names a `*-windows-msvc*` target folds only its resource directory (the
`gcc` row too, since a `cc` that IS such a clang reaches it by its stem), and every
other GNU driver keeps its whole `-v` list — libstdc++ IS read, and two machines
can differ in it under one `--version`.
other GNU driver keeps its `-v` list — libstdc++ IS read, and two machines
can differ in it under one `--version` — **except the machine's own `/usr/local/include`
and `/usr/include`** (`SystemIncludeDirectories`, `toolchain-v4`). Those are glibc and
every `-devel` package; a clean preprocessed compile opens neither (strace-measured under
gcc 16.2, clang 22.1 and 20.1, C and C++), and gcc reads a header beneath them only to
print a diagnostic's source excerpt, which reaches stderr and never the object. Folding
them split two Fedora boxes with identical compiler rpms on every toolchain. A root is
recognised by what it RESOLVES to (`IToolchainHost::ResolveDirectory`) and by EQUALITY,
never containment: libstdc++ resolves to `/usr/include/c++/<v>`, NESTED under a dropped
root, and a prefix match would drop it on every real Linux host. A sysroot's `usr/include`
deliberately is not recognised, and neither is Debian's `/usr/include/<multiarch>`
(#1783): a root that cannot be classified stays folded, so the Fedora measurement is the
whole of the claim.
- **That makes the banner the whole of `cl`'s identity, so a FALLBACK banner is
`NoEvidence` under `cl` whatever its layout** — a row that folds no tree BY
CONSTRUCTION is judged as having none, or #195's "every MSVC is `cl`" comes back behind
Expand Down Expand Up @@ -4351,3 +4362,9 @@ translation unit and every test green.
deduplicated across machines that share a cache, so the same compile can be admitted twice.
- **[#1731](https://github.com/LASTRADA-Software/pace/issues/1731)** — admission test gaps
deferred by the #1728 reviews.
- **[#1783](https://github.com/LASTRADA-Software/pace/issues/1783)** — a GNU toolchain's
fingerprint still folds Debian and Ubuntu's `/usr/include/<multiarch>` (kernel headers,
glibc's architecture bits, `-dev` packages' architecture headers), so two such machines
with one compiler still split on their `-dev` sets. `SystemIncludeDirectories` drops only
the native `/usr/include` and `/usr/local/include`, by resolved EQUALITY; recognising the
multiarch directory needs the driver's own name for it, and a clang triple is not it.
6 changes: 3 additions & 3 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,7 @@ launcher's cache key is made of. Before `apps/pace-cc/`, `CompileCache/`.
later edit wins by its recorded BASELINE, never by comparing clocks (`BudgetOverrideApplier`).
- A node SERVES while it identifies its toolchains: the cheap half at startup, the walk on the heartbeat thread's
first round, registering NOTHING until the fingerprint is real (`ToolchainSurvey` travels beside the map, no default).
- A GNU fingerprint drops the machine's `/usr/include` by RESOLVED EQUALITY, never containment: libstdc++ is NESTED there.
- Fingerprint inputs are CAPTURED, never re-gathered: `FINGERPRINT-INPUTS` reads what the survey
published; ONE capture walk per process runs on a thread of its own, and no request starts one.
- A node never reports a child's `ru_maxrss` as a compile's peak (Linux folds the spawner's RSS into it): a
Expand Down Expand Up @@ -1486,9 +1487,8 @@ wrong quantity**. Record a table of conditions, not a number.
- **The receiver owes one thing back: state what would FALSIFY a claim BEFORE opening
the file to check it**, or reading for confirmation stops at the first line that
matches. **A relayed diagnosis is relayed code.**
- **It does NOT reach a measurement's CONDITIONS — pin those, do not point at them**,
and say *why* they are pinned, or the next cleanup reverses it: the corrected comment
looks like the defect.
- **It does NOT reach a measurement's CONDITIONS — pin those, do not point at them**, and say
*why* they are pinned, or the next cleanup reverses it: the corrected comment looks like the defect.

All three, with the measurements that produced them, are in
[`.agent/rules/compile-cache.md`](.agent/rules/compile-cache.md).
Expand Down
8 changes: 7 additions & 1 deletion docs/getting-started/distributed-compilation.md
Original file line number Diff line number Diff line change
Expand Up @@ -861,7 +861,13 @@ on nothing but an installed Visual Studio language pack (#1656). `clang-cl`, and
the driver names when asked (`-print-resource-dir`), because they borrow the VC
toolset and the SDK rather than owning them — so two machines with different
SDKs or VS languages installed still match. A clang targeting anything else
carries every directory its `-v` search list names, as GCC does. None of them
carries every directory its `-v` search list names, as GCC does — except the
machine's own `/usr/local/include` and `/usr/include`, which hold its libraries
(glibc, every `-devel` package, hand installs) rather than its compiler. On Fedora,
where this was measured, two machines therefore need the same compiler packages,
not the same `-devel` set. Debian and Ubuntu also list `/usr/include/<multiarch>`,
which is still folded, so there two machines still need matching `-dev` sets
([#1783](https://github.com/LASTRADA-Software/pace/issues/1783)). None of them
derives its fingerprint from `INCLUDE`, which is set per developer command
prompt and never inherited by a service — so a worker installed as a service
matches the launchers that talk to it.
Expand Down
14 changes: 13 additions & 1 deletion docs/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,19 @@ machine fetches.
the resource directory the driver names when asked (`-print-resource-dir`). They
*borrow* the VC toolset and the SDK rather than owning them, so two machines with
different SDKs or VS languages installed still match. A clang targeting anything
else carries every directory its `-v` search list names, as GCC does.
else carries every directory its `-v` search list names, as GCC does — except
the machine's own `/usr/local/include` and `/usr/include`.
- **GCC and clang** leave out the machine's own `/usr/local/include` and
`/usr/include`, wherever a leading `/` names the filesystem root (measured on
Fedora Linux only). Those hold glibc and the headers of every `-devel` package
and hand install on the machine, not the compiler, and a worker reads neither to
generate code for a preprocessed compile. Folded, they split two Fedora machines
with byte-identical compiler packages on which libraries each had installed.
libstdc++ and the compiler's own `include` directory stay in, as does a sysroot's
`usr/include` — and so does Debian and Ubuntu's `/usr/include/<multiarch>`,
which holds architecture-specific library headers, so two such machines still
need matching `-dev` sets
([#1783](https://github.com/LASTRADA-Software/pace/issues/1783)).

None of them reads `INCLUDE` for its fingerprint. That variable is set by a
developer command prompt and is never inherited by a Windows service — so a worker
Expand Down
23 changes: 20 additions & 3 deletions docs/tools/pace-node.md
Original file line number Diff line number Diff line change
Expand Up @@ -444,7 +444,12 @@ change, so it costs the fleet no re-registration.

Two consequences worth knowing. An operator's pinned `<fingerprint>=<compiler>` is
**never** re-derived: it is not probed in the first place, and pinning a digest by
hand is how you force a fleet to agree while a machine is being repaired. And a
hand is how you force a fleet to agree while a machine is being repaired. A pin
is also never re-derived across an UPGRADE that changes how fingerprints are
computed: after moving to the `toolchain-v4` schema, a pin written under v3
advertises a digest no client computes any more, and that toolchain is silently
never matched. Re-derive such a pin (`pace-cc --print-toolchain-fingerprint
<compiler>`) or remove it. And a
machine whose only compiler an upgrade removed or broke keeps running while serving
nothing, rather than exiting — the compiler may come back with the next package, and
a routine upgrade must not be able to take a machine out of the fleet permanently.
Expand Down Expand Up @@ -520,7 +525,17 @@ IntelliSense XML lands in the toolset's `include` directory (#1656). So two
Windows machines need the same compiler *version*, not the same SDK or the same
Visual Studio languages. The banner names the target as well as the version —
`... Version 19.51.36252 for x64` — which is what tells the x86 and x64 `cl.exe`
of one toolset apart. Matching is
of one toolset apart. Under GCC and Clang the machine's own `/usr/local/include`
and `/usr/include` are left out for the same reason: they hold glibc and every
`-devel` package and hand install, not the compiler. libstdc++ and the compiler's
own `include` directory stay in, and so does every other directory the driver
lists. On a distribution that keeps its architecture headers in `/usr/include`
itself — Fedora, where this was measured — two machines therefore need the same
compiler packages rather than the same set of libraries. Debian and Ubuntu also
list `/usr/include/<multiarch>` (kernel headers, glibc's architecture bits, the
architecture-specific headers of `-dev` packages), which is still folded, so
there two machines still need matching `-dev` sets
([#1783](https://github.com/LASTRADA-Software/pace/issues/1783)). Matching is
byte-identical and cannot be loosened: an over-strict match costs a local
compile, an over-loose one produces a silently wrong object that is then stored
under a key other machines fetch.
Expand All @@ -529,7 +544,9 @@ Computing it walks that tree, which takes a few seconds the first time a machine
sees a toolchain and is cached afterwards.

**Which tree, per driver.** GCC and Clang are asked directly (`-E -v`) and fold
every directory they list — libstdc++, an Xcode SDK. A Clang whose target is
every directory they list — libstdc++, an Xcode SDK — except the machine's own
`/usr/local/include` and `/usr/include`, recognised by the directory they resolve
to. A Clang whose target is
`*-windows-msvc*` folds only its own resource directory, as `clang-cl` does
(`-print-resource-dir`): `<prefix>\lib\clang\<version>\include`. `cl` folds **no**
tree: its banner names its toolset version and target, and everything its layout
Expand Down
12 changes: 10 additions & 2 deletions src/Pace/CompileCache/ToolchainFingerprint.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,13 @@ struct ToolchainFile
/// own resource directory (`FingerprintTree`), so the same toolchain digests
/// differently under v2 and v3. The fingerprint cache's stamp folds this tag, so an
/// entry written under v2 is a miss rather than a v2 value served as a v3 one.
inline constexpr std::string_view FingerprintSchema = "toolchain-v3";
///
/// v4: a GNU-family driver no longer folds the machine's own `/usr/local/include` and
/// `/usr/include` (`SystemIncludeDirectories` in the launcher), which split machines
/// with byte-identical compiler packages on which `-devel` packages each had installed.
/// The same toolchain digests differently under v3 and v4, so a v3 node and a v4 node
/// match nothing until both are upgraded -- the safe direction, a local compile.
inline constexpr std::string_view FingerprintSchema = "toolchain-v4";

/// Everything a toolchain fingerprint is computed from, as the digest saw it.
///
Expand Down Expand Up @@ -133,7 +139,9 @@ struct FingerprintInputs
/// BORROWS (the VC toolset and the Windows SDK, under `cl` and under any clang
/// targeting MSVC) is never passed, because a worker opens no header from it for a
/// preprocessed compile, and passing it split machines on an installed VS language
/// pack's IntelliSense XML (#1656). A banner alone is what the cache key
/// pack's IntelliSense XML (#1656). Nor is the machine's own `/usr/include` or
/// `/usr/local/include` under a GNU driver (v4): glibc and every `-devel` package, which
/// split machines on what was installed BESIDE the compiler. A banner alone is what the cache key
/// already uses, and it is too weak here: two machines can print an identical
/// `--version` while resolving different libstdc++ headers, and for *distribution*
/// that produces a silently wrong object rather than the stale path a replay guard
Expand Down
3 changes: 2 additions & 1 deletion src/Pace/Distributed/FingerprintDiff_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
#include <catch2/catch_test_macros.hpp>

#include <algorithm>
#include <format>
#include <string>
#include <utility>
#include <vector>
Expand Down Expand Up @@ -153,7 +154,7 @@ TEST_CASE("Diff names a schema and a grammar difference as facts", "[distributed
CHECK(diff.schemaDiffers);
REQUIRE(diff.grammar.has_value());
CHECK(Unwrap(diff.grammar).second == "msvc");
CHECK(diff.verdict.contains("the fingerprint schemas differ (toolchain-v3 vs toolchain-v2)"));
CHECK(diff.verdict.contains(std::format("the fingerprint schemas differ ({} vs toolchain-v2)", Cc::FingerprintSchema)));
CHECK(diff.verdict.contains("the driver grammars differ (gnu vs msvc)"));
CHECK_FALSE(diff.inferred);
}
Expand Down
17 changes: 10 additions & 7 deletions src/apps/pace-cc/CmdLine.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -406,10 +406,12 @@ namespace
// version string covers x86_64 and aarch64 alike, and an arch-independent
// translation unit keys the same on both.
.targetDiscovery = TargetDiscovery::GnuTargetLine,
// No gcc targets MSVC, so this keeps every root for gcc itself. It is the
// Clang row's answer because a `cc`/`c++` that IS an MSVC-targeting clang
// reaches this row by its stem, and must fingerprint as the Clang row does.
.fingerprintTree = FingerprintTree::OwnTreeWhenTargetingMsvc,
// No gcc targets MSVC, so for gcc itself this keeps every root but the
// machine's `/usr/local/include` and `/usr/include`: libstdc++ and gcc's own
// `include` stay folded. It is the Clang row's answer because a `cc`/`c++`
// that IS an MSVC-targeting clang reaches this row by its stem, and must
// fingerprint as the Clang row does.
.fingerprintTree = FingerprintTree::OwnTrees,
.dispatchMacros = DispatchMacros::Expanded,
.includeProbeFlags = GnuIncludeProbe,
.targetProbeFlags = GnuTargetProbe,
Expand All @@ -428,9 +430,10 @@ namespace
.targetDiscovery = TargetDiscovery::ClangDriverLine,
// On Windows its `-v` list is clang's resource directory plus the VC
// toolset and the SDK it borrows, which it is identified without, as
// `clang-cl` is; anywhere else the list is libstdc++ or an SDK it reads,
// and stays folded (#1656).
.fingerprintTree = FingerprintTree::OwnTreeWhenTargetingMsvc,
// `clang-cl` is (#1656); anywhere else the list is libstdc++ or an SDK it
// reads, and stays folded -- all but the machine's own `/usr/local/include`
// and `/usr/include`, which hold its libraries rather than its compiler.
.fingerprintTree = FingerprintTree::OwnTrees,
.dispatchMacros = DispatchMacros::Expanded,
.includeProbeFlags = GnuIncludeProbe,
.targetProbeFlags = GnuTargetProbe,
Expand Down
25 changes: 18 additions & 7 deletions src/apps/pace-cc/CmdLine.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -230,19 +230,30 @@ enum class FingerprintTree : std::uint8_t
/// discovery finds -- the toolset it lives in, the newest SDK -- is one a worker
/// never opens for a dispatched compile.
None,
/// The driver's own resource directory when it targets MSVC, exactly as
/// `clang-cl` is identified -- and every discovered root otherwise. A GNU-grammar
/// clang on Windows lists the VC toolset and the SDK in its `-v` search list; on
/// Linux it lists libstdc++, which it does read, and that stays folded. Decided by
/// the target the include probe's own output names, so it costs no spawn on a
/// driver that targets anything else.
/// The trees the toolchain OWNS, out of a GNU-family `-v` search list.
///
/// When the driver targets MSVC, its own resource directory, exactly as `clang-cl`
/// is identified: a GNU-grammar clang on Windows lists the VC toolset and the SDK
/// it borrows. Decided by the target the include probe's own output names, so it
/// costs no spawn on a driver that targets anything else.
///
/// Otherwise every discovered root but the MACHINE's own include directories --
/// `SystemIncludeDirectories`, `/usr/local/include` and `/usr/include`, recognised
/// by what they resolve to. Those hold glibc and every `-devel` package and hand
/// install, never the compiler, so two machines with one compiler package split on
/// which libraries were installed beside it. libstdc++ and the compiler's resource
/// directory are listed as roots of their own and stay folded, as does any root
/// that cannot be classified. A SYSROOT's `usr/include` is deliberately not
/// recognised: a driver reads more of a sysroot than its headers -- Apple clang
/// derives `-target-sdk-version` from the SDK's `SDKSettings.json` -- so its tree
/// is the only evidence of which SDK a worker would compile against.
///
/// The `gcc` row carries it too, although no gcc targets MSVC. One binary reaches
/// both rows -- the node and `--print-toolchain-fingerprint` classify a driver by
/// its STEM, the launcher corrects that by its banner, and both share one cache
/// file per grammar -- so a `cc` that is really an MSVC-targeting clang must
/// fingerprint alike under either row, or a worker and its clients split.
OwnTreeWhenTargetingMsvc,
OwnTrees,
};

/// True when two family sets overlap.
Expand Down
25 changes: 25 additions & 0 deletions src/apps/pace-cc/ToolchainHost.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,31 @@ namespace
return std::filesystem::is_directory(*directory, ec) && !ec;
}

std::optional<std::string> ResolveDirectory(std::string_view path) override
{
auto const opened = PathFromNarrowText(path);
if (!opened.has_value())
return std::nullopt;

// `canonical`, never `weakly_canonical`: a path that does not exist must be no
// answer rather than its own lexical normalization, or two missing spellings
// that normalize alike would be reported as one directory.
std::error_code ec;
auto const resolved = std::filesystem::canonical(*opened, ec);
if (ec || !std::filesystem::is_directory(resolved, ec) || ec)
return std::nullopt;

// `PathBytes`, never `generic_string()`, which THROWS on Windows for a name
// holding an unpaired surrogate. The native separator is then folded to `/`
// by hand, which is all `generic_string()` added -- and only where it IS a
// separator: on POSIX a `\` is a byte of a name, and folding it would make
// `/usr\include` the same directory as `/usr/include`.
auto bytes = PathBytes(resolved);
if constexpr (std::filesystem::path::preferred_separator != '/')
std::ranges::replace(bytes, static_cast<char>(std::filesystem::path::preferred_separator), '/');
return bytes;
}

bool ExecutableExists(std::string_view path) override
{
auto const opened = PathFromNarrowText(path);
Expand Down
Loading
Loading