Repository navigation
perf(input): take each module from compileToModule in the bitcode format - #274
Draft
SizzleUnrlsd wants to merge 9 commits into
Draft
SizzleUnrlsd wants to merge 9 commits into
SizzleUnrlsd wants to merge 9 commits into
Conversation
The pinned commit, from CoreTrace/coretrace-compiler#265, gives the module of a compilation in the caller's context. The analyzer does not use it yet. Refs #273
…lper The lock, the move to the compile command's directory and the timing of a compilation now wrap a callable, which compile() is the only one to use yet. No behaviour changes. Refs #273
The bitcode format compiled each source to a temporary .bc file, read it into a string, copied it into a buffer and parsed it. The module now comes from compileToModule, created in the context of the load: nothing is written to a temporary file, and nothing is parsed. With the compile IR cache, the cache stores the bitcode of that module, written with the order of each value's uses. With --timing, a source compiled that way logs "Compiled to a module in N ms", where it logged "Compilation done", then "Bitcode parse done". check_compile_ir_format_switch recognized the bitcode format by the parse line: it now expects the new line from an empty cache, and either line once the cache is filled. Refs #273
| auto compileInWorkingDir = | ||
| [&](const std::vector<std::string>& compileArgs, | ||
| compilerlib::OutputMode outputMode) -> std::optional<compilerlib::CompileResult> | ||
| auto inWorkingDir = [&](auto&& compileHere) -> std::optional<compilerlib::CompileResult> |
| auto compileInWorkingDir = | ||
| [&](const std::vector<std::string>& compileArgs, | ||
| compilerlib::OutputMode outputMode) -> std::optional<compilerlib::CompileResult> | ||
| auto inWorkingDir = [&](auto&& compileHere) -> std::optional<compilerlib::CompileResult> |
The analyses walk a module read from the compile IR cache as they walked the compiled one only if the bitcode holds the order of each value's uses: the first match of a walk and the order of a variable's reports follow that order. Refs #273
…f uses A source compiled in the bitcode format is analyzed as the module of its compilation, then read from the compile IR cache by the next run. The test compiles a source whose calls Clang emits in another order than the module lists their functions, and compares each value's users between the two modules. A third load, on the entry written again without the order of uses, gives another order: the source tells the two apart, and a load reads the entry. Refs #273
The pinned commit was one of the branch of CoreTrace/coretrace-compiler#265. v0.13.0 is the release that holds compileToModule. Refs #273
SizzleUnrlsd
added a commit
that referenced
this pull request
Nov 18, 2026
In the second measure, the phases after loading took longer with #274 alone than on main, and the peak of memory was higher. #274 also moves coretrace-compiler from v0.10.0 to v0.13.0. Three analyzers tell the two effects apart: main, main built with that compiler, and main merged with #274. One value of --jobs is enough: main loads its inputs one after the other, whatever --jobs. Refs #274
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In the
bcformat of--compile-ir-format, the default, the analyzer compiled each source to a temporary.bcfile, read the file into a string, copied the string into a buffer and parsed it (#273). Each module now comes from the compilation itself, through coretrace-compiler'scompileToModule(CoreTrace/coretrace-compiler#265): no bitcode is written to a temporary file, and none is parsed.Closes #273
Change
build(deps): pin coretrace-compiler to compileToModule, thenbuild(deps): update coretrace-compiler to v0.13.0: the pin is the commit of the v0.13.0 release (e213053), which bringscompileToModule(feat(compilerlib): compile to an llvm::Module in the caller's context coretrace-compiler#265).perf(input): take each module from compileToModule in the bitcode format:--timing, a source compiled that way logsCompiled to a module in N ms. It loggedCompilation done in N ms, thenBitcode parse done in N ms. A module read from the compile IR cache is parsed as before, and still logsBitcode parse done in N ms.docs(readme)anddocs(changelog).docs(input): say why the cache's bitcode keeps the order of uses: a comment, after the review of the walks below.test(unit): check that a module read from the cache keeps the order of uses:testCompileIRCacheKeepsUseOrder, described under The test.Use lists: a point for the review
A module taken from the compilation lists each value's uses in the order the frontend created them. The module db7fb25 parsed from the
.bcfile listed them in the order the bitcode reader rebuilt them. The analyses walk a value's uses or users in 40 places, in 16 files, and a block's predecessors, which are the uses of the block, in 6 places. Line numbers below are those of main at 1fadba2; this branch does not change these files.What differs from db7fb25
Counted with a probe build that is not in this branch: each compiled module is written as bitcode without the order of uses, parsed, and compared with the compiled one, value by value.
The diagnostics and the functions reported are the same as db7fb25's on these three sets of sources, and the regression tests pass.
The walks where the order can change a result
14 of the 46 walks:
IRValueUtils.cpp:202,deriveAllocaNameParameterDebugBinding.cpp:38,spilledParameterStackBufferAnalysis.cpp:966,collectAccessesInvalidBaseReconstruction.cpp:326,:365,:505,:653,:675AllocaUsage.cpp:52;IntegerOverflowAnalysis.cpp:450,:483smt/SmtEncoding.cpp:812, predecessorsStackPointerEscape.cpp:1221StackBufferAnalysis.cpp:966, with the loadsFunctionFacts.cpp:195(loadsReadingStore) gives itNothing sorts the diagnostics of a report, and their ids follow their order.
The other 32 walks give the same result in any order. They look for a unique store and give up on a second one, require every use to pass or stop at the first that does with no budget, merge what every use gives, or fill a list that its reader sorts (
ResourceLifetimeAnalysis.cpp:768) or reads whole. Two walks look for a debug intrinsic among the uses,UninitializedVarAnalysis.cpp:1166and a branch ofderiveAllocaName: they find none, as a debug intrinsic holds its variable as metadata, which is not a use.Every use list reversed
A second mode of the probe build reverses the uses of every global variable, function, argument, block and instruction of each loaded module, before the analysis:
run_test.py --jobs 2 --no-cache: 2760/2760.StackPointerEscape: the walk ofStackPointerEscape.cpp:1221. TwoCommandInjection.NonLiteralCommanddiagnostics change places from run to run with any build, reversed or not.So the 14 walks depend on the order only for code that the tests and these sources rarely have, and where they do, the reversal changed the order of diagnostics, not the diagnostics.
What the cache keeps
In db7fb25, a run that filled the cache and a run that read it parsed the same bitcode, and walked the same order. Here, the run that fills the cache analyzes the compiled module, and a run that reads it analyzes a parsed one. The cache's bitcode is written with
ShouldPreserveUseListOrder, which stores the order of each value's uses for the reader to restore.The probe build writes each compiled module both ways, parses it, and compares it with the compiled module: the users of every global variable, function, argument, block and instruction, with the operand of each use.
The 75 lists that still differ
They are not another order. Each is a global variable named
.constantthat has two users in the compiled module, a store and a constant struct that nothing uses, and one in the parsed module, the store. For the 75, the users that are instructions are the same, in the same order.Where they are: one per call to
coretrace::log. The 53 sources hold 75 such calls, in 8 files, and each file has as many of these lists as calls: 37 inAnalyzerApp.cpp, 25 inInputPipeline.cpp, 4 inmain.cpp. The logger'sLogEntrytakesstd::source_location::current()as a default argument, and.constantis the record that the location points to.Where the struct comes from: as Clang's sources read, it evaluates the call to
current(), aconstevalfunction, to a constant struct that holds the address of.constant(AggExprEmitter::VisitConstantExpr), then stores that struct field by field (CreateCoercedStore). The struct stays in the LLVM context, among the users of.constant, used by nothing. Bitcode holds the constants a module uses: the writer leaves this one out, with or without the order of uses, and the module db7fb25 parsed never had it.A minimal source, in C++20:
Compiled,
.constanthas the store and the unused struct as users. Read from the cache, it has the store.loadsReadingStoredoes not meet them. It walks the users of the address a store writes to, and no store writes to.constant, a constant. The probe build counts 383,929 calls on the 53 sources and 21,904 on the 8 C++ files, none of them on a global variable. On Lua it counts 21,516 calls, 7 of them on a global variable, none of which has a constant among its users.It can meet one in another source, and gives the same loads. Take a
constevalfunction that returns a struct holding the address of a global variablegp, used as a default argument, and a function that writesgp. The walk meets the unused struct among the users ofgpin the run that compiles, and not in the run that reads the cache. It keeps the users that are loads and asks MemorySSA about each, so a constant is passed over. Both runs give the same report, byte for byte.The other walks: a walk that starts from a global variable or a function can meet such a constant, in the compiled module only. A mode of the probe build gives every global variable, then every defined function, of each loaded module an unused constant struct as a user.
run_test.pypasses 2760/2760 both ways, the reports of the 8 C++ files and of the 53 sources are the same byte for byte, and Lua's hold the same diagnostics and functions.Function::hasAddressTaken, whichStackPointerEscapeResolver.cpp:115asks, takes any user that is not a call for a taken address, such a constant included. No function has one on the three sets of sources.Not done:
removeDeadConstantUsers()on the globals of a compiled module would remove these constants, and the two modules would then list the same users. Nothing observed depends on them, so this pull request does not add it.Reports from an empty cache and from a filled one
This branch at dbe9c03,
--jobs=1, one compile IR cache directory per set of sources, empty before the first run:For Lua, the content of the diagnostics is the same in both runs: compared without their ids and in a common order, the 1024 diagnostics are the same, and so are the 2170 functions. Their order can vary. Two
CommandInjection.NonLiteralCommanddiagnostics change places from one run to the next, with any build, and the ids follow the order. The cache is not what moves them: five runs that read the same filled cache give both orders. Lua's reports are compared by content, and are not the same byte for byte.Not compared here: runs with more than one job. With #272, the comparison is to be made at one and four jobs, once coretrace-compiler holds both changes.
The test
testCompileIRCacheKeepsUseOrder, in the unit tests, takes the path of a run that fills the cache, then of a run that reads it:loadModuleForAnalysis, in the bitcode format, with a cache directory. This gives the module of the compilation, and an entry in the cache.In the source,
earlycallslate, defined afterother, and the three callcallee. Clang emits the bodies in the order of the source, while the module listslate, created by the call inearly, beforeother. Bitcode follows the module's order, so a reader that is not told the order of uses lists the callers ofcalleeanother way.Step 4 keeps the test from passing for nothing: it fails if the source stops telling the two orders apart, or if a load stops reading the entry. With
ShouldPreserveUseListOrdertaken out ofInputPipeline.cpp, step 3 fails, and only it.The test checks the order of uses on one C source. It does not compare reports: the comparisons above were run by hand.
A
.llgiven as inputOther code writes the cache entry of a
.llinput, and this pull request does not change it. As on main, that entry's bitcode holds no order of uses, while the first run analyzes the module parsed from the text. For a.llof three functions whose callee is declared after its two callers, as Clang prints declarations, the module parsed from the text and the module parsed from such bitcode list the callers in another order. The 12.llfiles oftest/show no such difference.An existing test changes
check_compile_ir_format_switchrecognized thebcformat by the log line of the bitcode parse, which a compiled source no longer has. It changes in the commit that changes the log, and keeps its three checks, with one shared cache directory:bc, empty cacheBitcode parse done inis loggedCompiled to a module inis logged, and neither parse line isllIR parse done inis logged,Bitcode parse done inis notCompiled to a module inis not logged eitherbcagainBitcode parse done inis logged,IR parse done inis notBitcode parse done in(the cache's bitcode) orCompiled to a module in(a build without the cache) is logged,IR parse done inis notThe changed test fails on db7fb25, where the first run logs
Compilation done in, thenBitcode parse done in, and passes on this branch.Measurements
Measured on 2026-11-12: coretrace-stack-analyzer db7fb25, with coretrace-compiler built from its main branch (f0a22d9), against the same analyzer with this change, on CoreTrace/coretrace-compiler#265. The measured build predates the log line of this branch and the fix of that pull request for LLVM 16, which LLVM 20 does not compile. macOS arm64 (M3, 8 cores), LLVM 20.1.2,
--format=json --timing, no IR cache, an empty resource summary cache. Medians of 5 runs, 3 for the 53 sources; the builds alternate within each repetition.--jobs--jobs=1, takes 8 % less time on both sets of C++ sources and 15 % less on Lua's. Sources still compile one at a time here, whatever--jobssays: perf(input): compile sources in parallel, each in its directory #272 is the change that compiles them in parallel.--jobs=1).On Linux
Measured on 2026-11-17 on a GitHub runner (run 44158421994): ubuntu-24.04 on x86-64, 4 CPUs, g++ 13.3, LLVM 20, glibc 2.39, each run in a cgroup of 4 CPUs and 6 GiB without swap. main (588e800), which builds coretrace-compiler v0.10.0, against this branch (aa8b606), which builds v0.13.0.
--jobs=1,--format=json --timing, no IR cache, an empty resource summary cache. Medians of 3 runs; the builds alternate from one repetition to the next. The branch is measured as it is, without main's last three pull requests (#283, #284, #285).--jobs=1: 99.3 s on main against 100.6 s for the whole run, 90.6 s against 89.4 s for loading, 8.7 s against 11.2 s for the phases after loading, 1009 MiB against 1142 MiB.--jobs=2and4, the branch alone still loads on worker threads, which main stopped doing in fix(app): load the shared modules one at a time, on the calling thread #285: those runs say nothing of the merged result and are not shown.Tests
Local runs, on macOS arm64 with AppleClang and LLVM 20.1.2, this branch built as a user builds it, coretrace-compiler fetched at the pinned commit, so that the compile IR cache is in use:
run_test.py --jobs 2 --no-cache, which runs the analyzer on the fixtures: 2760/2760 at dbe9c03, and at the three commits that change what is built, on main at 1fadba2: the pin alone (ae23d0b), the helper (3bbb15c) and the module path (02732dd). The script counts each unit test program below as one of the 2760, and as passed when the program is missing.-DBUILD_ANALYZER_UNIT_TESTS=ON). At dbe9c03 both were built and executed:stack_usage_analyzer_unit_testsends withAll analyzer module unit tests passed.after 399 checks, andownership_engine_unit_testswithAll ownership engine unit tests passed.after 116 checks. At ae23d0b, 3bbb15c and 02732dd they were not built here, so not executed. The CI builds both and requires them at every push.ShouldPreserveUseListOrdertaken out,the module read from the cache lists each value's uses as the compiled module doesfails, and the two other checks pass.--compile-ir-format=bc did not take the module from the compilation, and passes at 02732dd. Its three runs then logCompiled to a module in, thenCompilation done inandIR parse done in, thenCompilation cache hitandBitcode parse done in.scripts/format-check.shwith clang-format 20,scripts/check-license-compliance.shand the commit checker pass.CI at aa8b606
The head adds two commits to dbe9c03: the pin of v0.13.0 (6f50b2b) and its sentence in the changelog (aa8b606). They were not built locally: the runs below are their validation, on the merge of this branch into main at 588e800.
All ... unit tests passed.line. The macos-latest job got no runner at first, and the run was started again.Merging
coretrace-compiler v0.13.0 brings the changes that this branch and #272 need, and this branch pins it. #272 also changes how
InputPipeline.cppruns a compilation: it is being reworked, and will have this branch merged in.