Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,13 @@ Notable changes are grouped by release. See [all releases](https://github.com/Co
assigned from one another in a cycle that reaches two different arrays now resolve to no
array, as any pointer with several possible arrays does, where each used to be reported
against its own array.
- With the compile IR cache, a `.ll` input read from the cache lists each value's uses in the
order the run that parsed its text had (#279). The entry's bitcode holds that order; it was
written without it, and the two runs could list the callers of a function, or the users of
a global variable, another way. No diagnostic was seen to change. The cache schema changes
to `compile-ir-cache-v5`: entries written by an earlier build are no longer looked up, so
the first run after the update compiles or parses each input once more and writes its
entry again. The earlier entries stay in the cache directory, unused.
- Added an authors file and linked the repository documents from the README.
- SARIF rules carry a title (`shortDescription`) and a description (`fullDescription`) instead
of their id, and `describeRule` (`RuleCatalog.hpp`) gives them to programs linking the analyzer.
Expand Down
11 changes: 9 additions & 2 deletions src/analysis/InputPipeline.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,9 @@ namespace ctrace::stack::analysis
// v4: entries record the compiler that produced them (below). Builds of v3 ignore those
// fields and would reuse such an entry; under v4, they do not even look it up, since the
// key holds the schema.
constexpr llvm::StringLiteral kCompileIRCacheSchema = "compile-ir-cache-v4";
// v5: the entry of a `.ll` input holds the order of each value's uses (#279). An entry of
// v4 does not, and a build of v5 does not look it up: the first run writes it again.
constexpr llvm::StringLiteral kCompileIRCacheSchema = "compile-ir-cache-v5";

// The compiler that produced an entry: compile IR depends on it as much as on the
// command. An entry another coretrace-compiler revision or LLVM version produced is
Expand Down Expand Up @@ -1421,9 +1423,14 @@ namespace ctrace::stack::analysis
std::string sourceIR;
(void)readTextFile(inputPath, sourceIR);

// The order of each value's uses is written too. Some analyses depend on it:
// the first match of a walk, the order of a variable's reports. The reader
// otherwise rebuilds that order its own way, and a run that reads the cache
// would not walk the uses as the run that parsed the text did.
std::string llvmBitcode;
llvm::raw_string_ostream bitcodeStream(llvmBitcode);
llvm::WriteBitcodeToFile(*result.module, bitcodeStream);
llvm::WriteBitcodeToFile(*result.module, bitcodeStream,
/*ShouldPreserveUseListOrder=*/true);
bitcodeStream.flush();

std::vector<FileSnapshot> dependencySnapshots;
Expand Down
143 changes: 143 additions & 0 deletions test/unit/analyzer_module_unit_tests.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@

#include <llvm/Analysis/MemorySSA.h>
#include <llvm/AsmParser/Parser.h>
#include <llvm/Bitcode/BitcodeWriter.h>
#include <llvm/IR/Constants.h>
#include <llvm/IR/DebugInfoMetadata.h>
#include <llvm/IR/Function.h>
Expand Down Expand Up @@ -3287,6 +3288,147 @@ namespace
return std::nullopt;
}

/// A `.ll` given as input is parsed from its text by the run that fills the compile IR
/// cache, and from the bitcode of its cache entry by the next run. Analyses walk the uses of
/// a value, and can keep the first match, stop on a budget or report in the order of the
/// walk: the module read from the cache must list each value's uses in the order the module
/// parsed from the text does.
///
/// `callee` is declared after its two callers, as Clang prints declarations. The text parser
/// meets its uses before its declaration; a bitcode reader knows every function before it
/// reads a body, and lists the callers another way unless it is told the order. The last
/// load checks it, on the entry written again without that order: it also shows that a load
/// reads the entry.
///
/// An entry without the order is what a build of the cache schema v4 wrote. Recorded under
/// that schema, it must not be reused: the load parses the text and writes the entry again.
bool testCompileIRCacheKeepsUseOrderOfIRInput(TestReport& report)
{
char rootTemplate[] = "/tmp/ctrace-ir-use-order-XXXXXX";
if (mkdtemp(rootTemplate) == nullptr)
{
report.expect(false, "Compile IR cache of a .ll input: no temporary directory");
return false;
}
const std::filesystem::path root(rootTemplate);
const std::filesystem::path source = root / "callers.ll";
std::ofstream(source) << "define i32 @first(i32 %x) {\n"
" %r = call i32 @callee(i32 %x)\n"
" ret i32 %r\n"
"}\n"
"\n"
"define i32 @second(i32 %x) {\n"
" %r = call i32 @callee(i32 %x)\n"
" ret i32 %r\n"
"}\n"
"\n"
"declare i32 @callee(i32)\n";

ctrace::stack::AnalysisConfig config;
config.compileIRCacheDir = (root / "cache").string();
const auto load = [&](LoadedModule& out)
{
llvm::SMDiagnostic err;
out.module = ctrace::stack::analysis::detail::loadModuleForAnalysis(
source.string(), config, out.context, err, "known")
.module;
return out.module != nullptr;
};
// The functions that call `callee`, in the order of its use list.
const auto callers = [](const llvm::Module& module)
{
std::vector<std::string> names;
if (const llvm::Function* callee = module.getFunction("callee"))
{
for (const llvm::User* user : callee->users())
{
if (const auto* call = llvm::dyn_cast<llvm::CallBase>(user))
names.push_back(call->getFunction()->getName().str());
}
}
return names;
};

LoadedModule parsed;
std::filesystem::path entry;
if (load(parsed))
{
std::error_code ec;
for (const auto& file : std::filesystem::directory_iterator(root / "cache", ec))
{
if (file.path().extension() == ".bc")
entry = file.path();
}
}
report.expect(
!entry.empty(),
"Compile IR cache of a .ll input: the first load writes the module's bitcode");
LoadedModule cached;
if (entry.empty() || !load(cached))
{
report.expect(false, "Compile IR cache of a .ll input: the second load gives a module");
std::filesystem::remove_all(root);
return false;
}

const std::vector<std::string> expected = callers(*parsed.module);
report.expect(expected.size() == 2,
"Compile IR cache of a .ll input: the parsed text lists the two callers");
report.expect(callers(*cached.module) == expected,
"Compile IR cache of a .ll input: the module read from the cache lists the "
"callers of a function as the parsed text does");

{
std::error_code ec;
llvm::raw_fd_ostream bitcode(entry.string(), ec);
if (!ec)
llvm::WriteBitcodeToFile(*parsed.module, bitcode);
}
LoadedModule unordered;
report.expect(load(unordered) && callers(*unordered.module) != expected,
"Compile IR cache of a .ll input: an entry written without the order of "
"uses lists the callers another way");

// The schema the entry records, which becomes `replacement` when one is given.
const std::filesystem::path meta = std::filesystem::path(entry).replace_extension(".json");
const auto schema = [&](const char* replacement) -> std::string
{
std::ostringstream text;
text << std::ifstream(meta).rdbuf();
llvm::Expected<llvm::json::Value> record = llvm::json::parse(text.str());
if (!record)
{
llvm::consumeError(record.takeError());
return {};
}
llvm::json::Object* object = record->getAsObject();
if (object == nullptr)
return {};
const std::string recorded = object->getString("schema").value_or("").str();
if (replacement != nullptr)
{
(*object)["schema"] = replacement;
std::string written;
llvm::raw_string_ostream out(written);
out << *record;
out.flush();
std::ofstream(meta, std::ios::trunc) << written;
}
return recorded;
};
const std::string current = schema("compile-ir-cache-v4");
LoadedModule rebuilt;
report.expect(load(rebuilt) && callers(*rebuilt.module) == expected,
"Compile IR cache of a .ll input: an entry of the schema that held no "
"order is not reused");
report.expect(!current.empty() && schema(nullptr) == current,
"Compile IR cache of a .ll input: the load writes that entry again, under "
"the current schema");

std::filesystem::remove_all(root);
return report.failures == 0;
}

/// The compile IR cache records the coretrace-compiler revision of the build. A build that
/// does not know it (coretrace-compiler.cmake) must neither read the cache, whose entries
/// could come from any compiler, nor write it, with entries no build could check.
Expand Down Expand Up @@ -3710,6 +3852,7 @@ int main(int argc, char** argv)
(void)testLocationResolver(repoRoot, report);
(void)testReachabilityService(repoRoot, report);
(void)testModulePreparationService(repoRoot, report);
(void)testCompileIRCacheKeepsUseOrderOfIRInput(report);
(void)testIntRangeFacts(repoRoot, report);
(void)testEveryReportedCweIsScored(repoRoot, report);
(void)testAnalysisReportContract(repoRoot, report);
Expand Down
Loading