Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion docs/tools/pace-cc.md
Original file line number Diff line number Diff line change
Expand Up @@ -713,6 +713,20 @@ than being read by position and misread. The report counts failed compiles only
records that carry an exit code, which is what tells a compiler killed from outside
apart from a cache that broke the build.

Three outcomes say a cache answer was not the healthy one it looked like:

- **`STORE-FAILED`**: compiled, and the cache did not take the object, because it refused or
could not be reached for the store. `MISS` means *compiled and stored*, so a cache out of
budget used to read as a cold one. The next build misses again.
- **`STALE`**: the cache had the object, but its dependency record no longer held (a replayed
header is gone, or the value carries no depfile the build asked for). The compile ran for real
and its store repairs the entry. It used to read as an ordinary `MISS`.
- **`VERIFY-FAILED`**: `PACE_VERIFY`'s fresh compile of a served hit failed. The served object is
kept for the build, but a source the compiler now rejects did not produce it, so the entry is
suspect.

`--show-stats` and the HTML report count each on its own line, and only when there is one.

`INPUTS-MOVED` is a compile one of whose inputs changed while the key was taken or during
the compile — an editor's save, a `git checkout` in another terminal — so the launcher did
not store its object (a direct-mode manifest built from the content the key was taken over
Expand Down Expand Up @@ -988,7 +1002,8 @@ retrying, the other never will be:
| *(silence)* | The cached object is the object this compiler produces. |
| `WRONG OBJECT served for key …` | It is not. The message names what differed — a section such as `.text$mn` is stale code; `.debug$S` or `.chks64` is a foreign build path. The fresh object was used, so this build is unaffected. Find the machine that stored it. The compile is recorded as `VERIFY-MISMATCH` rather than `HIT` — in `invocations.log`, on the `PACE_VERBOSE` trace line, and as `wrong object` in `--show-stats`, which rates the hit rate against it — because the build did not use what the cache served. |
| `NONDETERMINISTIC compiler for key …` | Not a cache fault. The served object differed from a fresh compile, so the launcher compiled a second time — and the two fresh objects differ from each other too: this compiler does not reproduce its own output for this translation unit (seen with `cl` on some units). Nothing is known about the cached object either way, and the entry is kept. Recorded as `HIT`. |
| `could not verify the hit for key …` | The check did not complete: the fresh compile failed, or a file could not be read. Nothing is known about the cached object either way, and the next hit may well answer. |
| `VERIFY FAILED for key …` | The fresh compile of this source failed, so the served object cannot be what this compiler produces from it. The served object is kept for this build. Find what changed in the source or the toolchain, and treat the entry as suspect. Recorded as `VERIFY-FAILED`. |
| `could not verify the hit for key …` | The check did not complete: a file could not be read, or the served object could not be put back after a failed fresh compile. Nothing is known about the cached object either way, and the next hit may well answer. |
| `cannot verify hits for this toolchain …` | This build cannot lay out the object format its own compiler produced, so it can say nothing about any hit — not this one and not the next. A property of the toolchain, not a statement about your cache. |

## Debug paths in a replayed object
Expand Down
5 changes: 5 additions & 0 deletions src/apps/pace-cc/HitVerification.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,11 @@ std::string DescribeVerdict(HitComparison const& comparison, std::string_view ke
return withDetail(std::format("pace-cc: cannot verify hits for this toolchain, so the hit for key "
"{} was neither confirmed nor faulted",
key));
case HitVerdict::FreshCompileFailed:
return withDetail(std::format("pace-cc: VERIFY FAILED for key {} -- the fresh compile of this source "
"failed, so the served object cannot be what this compiler produces from it; "
"the served object was kept for this build, and the entry is suspect",
key));
case HitVerdict::Nondeterministic:
// Not a finding about the cache either (#1758): the compiler disagreed with
// ITSELF, so the served object's difference proves nothing. Said, so a reader
Expand Down
4 changes: 4 additions & 0 deletions src/apps/pace-cc/HitVerification.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ enum class HitVerdict : std::uint8_t
/// that. A false `WRONG OBJECT` is the answer that gets a verifier switched off,
/// which is why this is its own state rather than `Mismatched`.
Nondeterministic,
/// Sampled, and the fresh compile of the same source FAILED (#1707). The served object is put
/// back, but a source this compiler rejects did not produce it, so this is evidence against
/// the entry -- never the `Inconclusive` it was folded into, which tallied it as a hit.
FreshCompileFailed,
};

/// Verification is off; no hit is checked.
Expand Down
10 changes: 10 additions & 0 deletions src/apps/pace-cc/HitVerification_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -393,3 +393,13 @@ TEST_CASE("A verification put-back keeps the served object aside until it is con
CHECK(line.contains("replace"));
}
}

TEST_CASE("A fresh compile that failed is described as evidence against the entry", "[launcher][verify][1707]")
{
auto const line = DescribeVerdict(
{ .verdict = HitVerdict::FreshCompileFailed, .comparison = std::nullopt, .detail = "the fresh compile exited 2" },
"k");
CHECK(line.contains("VERIFY FAILED"));
CHECK(line.contains("exited 2"));
CHECK_FALSE(line.contains("could not verify"));
}
55 changes: 53 additions & 2 deletions src/apps/pace-cc/Stats.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,9 @@ namespace
case Outcome::Miss:
case Outcome::VerifyMismatch:
case Outcome::InputsMoved:
case Outcome::StoreFailed:
case Outcome::Stale:
case Outcome::VerifyFailed:
return true;
case Outcome::Uncacheable:
case Outcome::Unavailable:
Expand All @@ -536,6 +539,12 @@ namespace
std::uint64_t verifyMismatches {};
/// Compiles an input changed during; see `Outcome::InputsMoved`.
std::uint64_t inputsMoved {};
/// Compiles whose STORE was not taken; see `Outcome::StoreFailed`.
std::uint64_t storeFailures {};
/// Hits refused as stale and recompiled; see `Outcome::Stale`.
std::uint64_t staleHits {};
/// Verified hits whose fresh compile failed; see `Outcome::VerifyFailed`.
std::uint64_t verifyFailures {};
/// Records `CountsAsServable` admits; read through `Servable()`.
std::uint64_t servableCount {};

Expand Down Expand Up @@ -575,7 +584,8 @@ namespace

[[nodiscard]] std::uint64_t Total() const noexcept
{
return hits + misses + uncacheable + unavailable + verifyMismatches + inputsMoved;
return hits + misses + uncacheable + unavailable + verifyMismatches + inputsMoved + storeFailures + staleHits
+ verifyFailures;
}

/// The compiles the cache was ASKED for, which is what a hit rate is rated against;
Expand Down Expand Up @@ -670,6 +680,12 @@ namespace
return Outcome::VerifyMismatch;
if (token == "INPUTS-MOVED")
return Outcome::InputsMoved;
if (token == "STORE-FAILED")
return Outcome::StoreFailed;
if (token == "STALE")
return Outcome::Stale;
if (token == "VERIFY-FAILED")
return Outcome::VerifyFailed;
return Outcome::Unavailable;
}

Expand Down Expand Up @@ -1136,6 +1152,16 @@ namespace
out << " wrong object : " << Colorize(std::to_string(tally.verifyMismatches), palette.bad, palette.reset)
<< " (hits PACE_VERIFY rejected -- " << Colorize("WRONG OBJECT SERVED", palette.bad, palette.reset)
<< "; the fresh compile was used)\n";
if (tally.verifyFailures > 0)
out << " verify failed: " << Colorize(std::to_string(tally.verifyFailures), palette.bad, palette.reset)
<< " (hits PACE_VERIFY could not recompile -- the source no longer compiles, so the served object is "
"suspect)\n";
if (tally.staleHits > 0)
out << " stale hits : " << Colorize(std::to_string(tally.staleHits), palette.neutral, palette.reset)
<< " (entries whose dependency record no longer held; recompiled and re-stored)\n";
if (tally.storeFailures > 0)
out << " store failed : " << Colorize(std::to_string(tally.storeFailures), palette.bad, palette.reset)
<< " (compiled, and the cache did not take the object -- the next build misses again)\n";

if (!tally.reasons.empty())
{
Expand Down Expand Up @@ -1193,6 +1219,12 @@ std::string_view ToStringView(Outcome outcome) noexcept
return "VERIFY-MISMATCH";
case Outcome::InputsMoved:
return "INPUTS-MOVED";
case Outcome::StoreFailed:
return "STORE-FAILED";
case Outcome::Stale:
return "STALE";
case Outcome::VerifyFailed:
return "VERIFY-FAILED";
case Outcome::Unavailable:
break;
}
Expand All @@ -1203,6 +1235,8 @@ Outcome OutcomeOfServedHit(HitVerdict verdict) noexcept
{
switch (verdict)
{
case HitVerdict::FreshCompileFailed:
return Outcome::VerifyFailed;
case HitVerdict::Mismatched:
return Outcome::VerifyMismatch;
case HitVerdict::NotChecked:
Expand Down Expand Up @@ -1441,6 +1475,17 @@ namespace
case Outcome::InputsMoved:
++tally->inputsMoved;
break;
case Outcome::StoreFailed:
++tally->storeFailures;
tally->missMs.push_back(record.elapsedMs);
break;
case Outcome::Stale:
++tally->staleHits;
tally->missMs.push_back(record.elapsedMs);
break;
case Outcome::VerifyFailed:
++tally->verifyFailures;
break;
}
if (CountsAsServable(record.outcome))
++tally->servableCount;
Expand Down Expand Up @@ -1469,7 +1514,7 @@ namespace
// uncacheable file is visible rather than hidden in a percentage, and so
// is a file an edit raced, which names the source an editor touched.
if ((record.outcome == Outcome::Uncacheable || record.outcome == Outcome::Unavailable
|| record.outcome == Outcome::InputsMoved)
|| record.outcome == Outcome::InputsMoved || record.outcome == Outcome::StoreFailed)
&& !record.source.empty())
++folded.neverCached[record.source];
}
Expand Down Expand Up @@ -1972,6 +2017,12 @@ std::string FormatHtmlReport(std::string_view groupFilter)
// Conditional for the same reason: most dashboards have no editor racing a build.
if (overall.inputsMoved > 0)
AppendTallyCard(out, "inputs moved", overall.inputsMoved, "uncache");
if (overall.verifyFailures > 0)
AppendTallyCard(out, "verify failed", overall.verifyFailures, "bad");
if (overall.staleHits > 0)
AppendTallyCard(out, "stale hits", overall.staleHits, "miss");
if (overall.storeFailures > 0)
AppendTallyCard(out, "store failed", overall.storeFailures, "bad");
out << "</div>";

out << R"(<div class="panel"><div class="panel-title">hit rate over time</div>)" << RenderTrendSvg(records) << "</div>";
Expand Down
14 changes: 14 additions & 0 deletions src/apps/pace-cc/Stats.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,20 @@ enum class Outcome : std::uint8_t
/// the path is on the `PACE_VERBOSE` line.
/// An input that could not be OBSERVED at all is `Uncacheable`, not this.
InputsMoved,
/// Compiled, and the STORE was attempted and not taken (#1707): the daemon refused it or
/// could not be reached for it. `Miss` claims "compiled and stored", so a store that ran out
/// of budget read as a healthy miss and the next build missed again for no visible reason.
StoreFailed,
/// A hit REFUSED as stale -- the replayed dependency record names a file that is gone, or the
/// value carries no depfile the build asked for -- so the compile ran for real and its STORE
/// repairs the entry (#1707). Logged as `Miss` it was indistinguishable from a cold cache, and
/// a cache serving stale entries read as one that was merely empty.
Stale,
/// Served, then `PACE_VERIFY`'s fresh compile of the same source FAILED (#1707). The served
/// object put back is the one the build uses, but a source the compiler now rejects cannot
/// have produced it, which is strong evidence the entry is wrong -- never the "inconclusive"
/// hit it was tallied as.
VerifyFailed,
};

/// @param outcome The outcome to name.
Expand Down
29 changes: 27 additions & 2 deletions src/apps/pace-cc/Stats_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,9 @@ TEST_CASE("ToStringView names every outcome by its token")
CHECK(ToStringView(Outcome::Unavailable) == "UNAVAILABLE");
CHECK(ToStringView(Outcome::VerifyMismatch) == "VERIFY-MISMATCH");
CHECK(ToStringView(Outcome::InputsMoved) == "INPUTS-MOVED");
CHECK(ToStringView(Outcome::StoreFailed) == "STORE-FAILED");
CHECK(ToStringView(Outcome::Stale) == "STALE");
CHECK(ToStringView(Outcome::VerifyFailed) == "VERIFY-FAILED");
}

TEST_CASE("Every outcome reads back from the log as itself", "[stats][1650]")
Expand All @@ -174,8 +177,10 @@ TEST_CASE("Every outcome reads back from the log as itself", "[stats][1650]")
// record carries. `Unavailable` is last on purpose -- it is also what an unknown token reads
// back as, so a token the parser lost would pass for it; every other entry cannot.
ScopedStateDir const scoped;
constexpr std::array outcomes { Outcome::Hit, Outcome::Miss, Outcome::Uncacheable, Outcome::VerifyMismatch,
Outcome::InputsMoved, Outcome::Unavailable };
constexpr std::array outcomes {
Outcome::Hit, Outcome::Miss, Outcome::Uncacheable, Outcome::VerifyMismatch, Outcome::InputsMoved,
Outcome::StoreFailed, Outcome::Stale, Outcome::VerifyFailed, Outcome::Unavailable
};
for (auto const outcome: outcomes)
AppendRecord(MakeRecord(outcome, "main", std::string { ToStringView(outcome) } + ".cpp", 10));
auto const records = ParseLog("");
Expand All @@ -193,6 +198,8 @@ TEST_CASE("A hit the verifier rejected is recorded as its own outcome and never
// fresh object: the cache's was wrong. Recording that as a HIT counted every wrong
// object the verifier caught as a success.
CHECK(OutcomeOfServedHit(HitVerdict::Mismatched) == Outcome::VerifyMismatch);
// A fresh compile that FAILED is evidence against the entry, never an inconclusive hit (#1707).
CHECK(OutcomeOfServedHit(HitVerdict::FreshCompileFailed) == Outcome::VerifyFailed);
// Every other verdict leaves the served object where the build uses it.
for (auto const verdict:
{ HitVerdict::NotChecked, HitVerdict::Matched, HitVerdict::Inconclusive, HitVerdict::Unsupported })
Expand Down Expand Up @@ -1276,3 +1283,21 @@ TEST_CASE("Stats.report-omits-wait-and-peak-without-readings", "[stats][log-form
CHECK_FALSE(report.contains("waited"));
CHECK_FALSE(report.contains("peak compile memory"));
}

TEST_CASE("A failed store, a stale hit and a failed verify are each named in the report", "[stats][1707]")
{
// #1707: each read as a healthy outcome -- a failed STORE as a MISS ("compiled and stored"),
// a refused stale entry as a cold MISS, a verify whose fresh compile failed as a HIT.
ScopedStateDir const scoped;
AppendRecord(MakeRecord(Outcome::Miss, "main", "a.cpp", 10));
AppendRecord(MakeRecord(Outcome::StoreFailed, "main", "b.cpp", 10));
AppendRecord(MakeRecord(Outcome::Stale, "main", "c.cpp", 10));
AppendRecord(MakeRecord(Outcome::VerifyFailed, "main", "d.cpp", 10));

auto const report = FormatReport("");
CHECK(report.contains("compiles : 4"));
CHECK(report.contains("misses : 1"));
CHECK(report.contains("store failed : 1"));
CHECK(report.contains("stale hits : 1"));
CHECK(report.contains("verify failed: 1"));
}
23 changes: 20 additions & 3 deletions src/apps/pace-cc/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1222,7 +1222,15 @@ void AddQueued(std::optional<std::uint64_t>& total, std::optional<std::uint64_t>
// failed verification never costs the build its object; and the aside, the only
// good copy, goes only once the put-back is confirmed.
auto const files = Cc::MakeDiskFiles();
return Cc::RestoreServedObject(aside, *served, *files);
auto restored = Cc::RestoreServedObject(aside, *served, *files);
// Put back cleanly: the compile's failure is the finding, and a source this compiler
// rejects did not produce the object served (#1707). A put-back that failed keeps its
// own inconclusive answer, which says where the served object went.
if (restored.detail.empty())
return { .verdict = Cc::HitVerdict::FreshCompileFailed,
.comparison = std::nullopt,
.detail = std::format("the fresh compile exited {}", run.exitCode) };
return restored;
}

auto comparison = Cc::CompareObjectFiles(aside, *served);
Expand Down Expand Up @@ -3281,6 +3289,9 @@ struct CachedCompile
// absent bound copy: one whose bound value is damaged, of another generation or stale reaches the
// STORE too, and a portable store there would overwrite the marker for every checkout.
std::optional<Cc::CheckoutParts> metMarker {};
// A hit the FETCH found and refused as stale, so the compile that follows is the repair of an
// entry rather than a cold miss, and its record says so (#1707).
bool staleHit { false };

/// @param result What `RunCached` returns.
/// @return A step that settled the invocation with @p result.
Expand Down Expand Up @@ -3660,6 +3671,7 @@ FetchStep CachedCompile::ServeHit(Cc::CacheOutcome const& outcome)
return FetchStep { .decided = true,
.result = Warn(record, Cc::CacheActionReason(Cc::FetchObservation::HitObjectUnwritable)) };
}
staleHit = materialized.disposition == HitDisposition::Stale;
// Stale: the object is fine but the dependency record it carries is not
// true here -- or the depfile this build names could not be written, which
// says nothing against the entry -- so fall through and compile for real. The STORE that follows
Expand Down Expand Up @@ -3705,8 +3717,10 @@ void CachedCompile::TraceMiss()
{
// Which parts, from the marker -- and no claim about WHICH of them differs, which
// this end cannot know: the trace and the tallied reason say the same thing.
TraceOutcome(
record, Cc::Outcome::Miss, key, boundMiss.has_value() ? Cc::BoundMissQualifier(*boundMiss) : std::string {});
TraceOutcome(record,
staleHit ? Cc::Outcome::Stale : Cc::Outcome::Miss,
key,
boundMiss.has_value() ? Cc::BoundMissQualifier(*boundMiss) : std::string {});
// A miss carrying a reason, as a crossed reply is: the cache answered honestly, and
// what it holds under this key names paths this compile does not share -- or this
// compile's own copy was evicted. Worth ranking, because a translation unit that is a
Expand Down Expand Up @@ -3893,6 +3907,9 @@ int CachedCompile::FinishMiss(Cc::CompileRun& run, bool& stored)
}

stored = StoreCompiledObject(std::move(*permit), record, cfg, key, value.objectBlob, encoded, checkout, metMarker);
// A compile the cache did not take is not the "compiled and stored" a MISS claims (#1707).
if (!stored && (record.outcome == Cc::Outcome::Miss || record.outcome == Cc::Outcome::Stale))
record.outcome = Cc::Outcome::StoreFailed;

// Record the direct-mode manifest so the NEXT compile of this TU can reach the
// object without preprocessing. The include set comes from the /showIncludes
Expand Down
Loading