Skip to content

perf(input): take each module from compileToModule in the bitcode format - #274

Draft
SizzleUnrlsd wants to merge 9 commits into
mainfrom
perf/compile-to-module
Draft

SizzleUnrlsd wants to merge 9 commits into
mainfrom
perf/compile-to-module

Conversation

@SizzleUnrlsd

@SizzleUnrlsd SizzleUnrlsd commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

In the bc format of --compile-ir-format, the default, the analyzer compiled each source to a temporary .bc file, read the file into a string, copied the string into a buffer and parsed it (#273). Each module now comes from the compilation itself, through coretrace-compiler's compileToModule (CoreTrace/coretrace-compiler#265): no bitcode is written to a temporary file, and none is parsed.

Closes #273

Change

  • build(deps): pin coretrace-compiler to compileToModule, then build(deps): update coretrace-compiler to v0.13.0: the pin is the commit of the v0.13.0 release (e213053), which brings compileToModule (feat(compilerlib): compile to an llvm::Module in the caller's context coretrace-compiler#265).
  • perf(input): take each module from compileToModule in the bitcode format:
    • the bitcode compilation gives its module, created in the context of the load; the temporary file, its cleanup, the read and the parse are gone;
    • with the compile IR cache, the cache stores the bitcode of that module;
    • with --timing, a source compiled that way logs Compiled to a module in N ms. It logged Compilation done in N ms, then Bitcode parse done in N ms. A module read from the compile IR cache is parsed as before, and still logs Bitcode parse done in N ms.
  • docs(readme) and docs(changelog).
  • docs(input): say why the cache's bitcode keeps the order of uses: a comment, after the review of the walks below.
  • test(unit): check that a module read from the cache keeps the order of uses: testCompileIRCacheKeepsUseOrder, described under The test.

Use lists: a point for the review

A module taken from the compilation lists each value's uses in the order the frontend created them. The module db7fb25 parsed from the .bc file listed them in the order the bitcode reader rebuilt them. The analyses walk a value's uses or users in 40 places, in 16 files, and a block's predecessors, which are the uses of the block, in 6 places. Line numbers below are those of main at 1fadba2; this branch does not change these files.

What differs from db7fb25

Counted with a probe build that is not in this branch: each compiled module is written as bitcode without the order of uses, parsed, and compared with the compiled one, value by value.

Sources Values with several uses In another order Of which
Lua 5.4.6, 34 C files 18,641 698 467 functions, 132 blocks, 99 global variables
8 C++ files 23,864 600 248 functions, 160 allocas, 104 other instructions, 56 blocks, 16 arguments, 16 global variables
This repository's 53 C++ sources 298,054 6,244 3,427 functions, 1,145 allocas, 620 other instructions, 570 blocks, 260 arguments, 222 global variables

The diagnostics and the functions reported are the same as db7fb25's on these three sets of sources, and the regression tests pass.

The walks where the order can change a result

14 of the 46 walks:

What the order can change Walk How
A selection IRValueUtils.cpp:202, deriveAllocaName An unnamed slot takes the name of the first named variable its address is stored in. The reports print that name.
ParameterDebugBinding.cpp:38, spilledParameter An argument takes the debug variable of the first slot, declared as a parameter, that it is stored in.
StackBufferAnalysis.cpp:966, collectAccesses An access reached through two pointer variables keeps the names of the last one, in the alias path of its report.
An early stop InvalidBaseReconstruction.cpp:326, :365, :505, :653, :675 The five walks spend one budget of 200,000 steps per function. Once it is spent, they stop with what they have reached.
AllocaUsage.cpp:52; IntegerOverflowAnalysis.cpp:450, :483 A depth limit, 20 and 32, with one set of visited values for all branches: a value first reached at the limit answers no when a shorter path reaches it again.
smt/SmtEncoding.cpp:812, predecessors The condition to reach a block is an Or of its predecessors' conditions, in their order: the same formula, written another way, for a solver with a time limit and a node budget.
The order of the diagnostics StackPointerEscape.cpp:1221 The escapes of one variable are reported in the order of the walk.
StackBufferAnalysis.cpp:966, with the loads FunctionFacts.cpp:195 (loadsReadingStore) gives it The accesses through one address are reported in the order of the walk.

Nothing sorts the diagnostics of a report, and their ids follow their order.

The other 32 walks give the same result in any order. They look for a unique store and give up on a second one, require every use to pass or stop at the first that does with no budget, merge what every use gives, or fill a list that its reader sorts (ResourceLifetimeAnalysis.cpp:768) or reads whole. Two walks look for a debug intrinsic among the uses, UninitializedVarAnalysis.cpp:1166 and a branch of deriveAllocaName: they find none, as a debug intrinsic holds its variable as metadata, which is not a use.

Every use list reversed

A second mode of the probe build reverses the uses of every global variable, function, argument, block and instruction of each loaded module, before the analysis:

  • run_test.py --jobs 2 --no-cache: 2760/2760.
  • The 8 C++ files and the 53 sources: the same reports, byte for byte. 23,840 and 297,749 values start with another user.
  • Lua: the same diagnostics and functions. Four of the 1024 diagnostics change places, two pairs of neighbours, all StackPointerEscape: the walk of StackPointerEscape.cpp:1221. Two CommandInjection.NonLiteralCommand diagnostics change places from run to run with any build, reversed or not.

So the 14 walks depend on the order only for code that the tests and these sources rarely have, and where they do, the reversal changed the order of diagnostics, not the diagnostics.

What the cache keeps

In db7fb25, a run that filled the cache and a run that read it parsed the same bitcode, and walked the same order. Here, the run that fills the cache analyzes the compiled module, and a run that reads it analyzes a parsed one. The cache's bitcode is written with ShouldPreserveUseListOrder, which stores the order of each value's uses for the reader to restore.

The probe build writes each compiled module both ways, parses it, and compares it with the compiled module: the users of every global variable, function, argument, block and instruction, with the operand of each use.

Sources Lists that differ, order left out Lists that differ, order stored
Lua 5.4.6, 34 C files 698 0
8 C++ files 600 0
This repository's 53 C++ sources 6,244 75
  • Kept, with the order stored: the users of every function, argument, block and instruction, in their order, on the three sets of sources. The same holds for every global variable of Lua and of the 8 C++ files, and for all but 75 of the 53 sources.
  • Lost, with the order left out: the lists of the middle column. In C++, they include those of allocas, arguments, other instructions and blocks, which the 14 walks start from. In C, they are those of functions, global variables and blocks, whose uses are their predecessors.
  • The cost: 0.7 % of the cache of the 8 C++ files (9.4 MB) and 2 % of Lua's (3.8 MB).
  • A limit of this comparison: a user that is a constant counts as a constant and an operand number. The order among several constant users of one value is not compared, nor are the use lists of the constants themselves.

The 75 lists that still differ

They are not another order. Each is a global variable named .constant that has two users in the compiled module, a store and a constant struct that nothing uses, and one in the parsed module, the store. For the 75, the users that are instructions are the same, in the same order.

  • Where they are: one per call to coretrace::log. The 53 sources hold 75 such calls, in 8 files, and each file has as many of these lists as calls: 37 in AnalyzerApp.cpp, 25 in InputPipeline.cpp, 4 in main.cpp. The logger's LogEntry takes std::source_location::current() as a default argument, and .constant is the record that the location points to.

  • Where the struct comes from: as Clang's sources read, it evaluates the call to current(), a consteval function, to a constant struct that holds the address of .constant (AggExprEmitter::VisitConstantExpr), then stores that struct field by field (CreateCoercedStore). The struct stays in the LLVM context, among the users of .constant, used by nothing. Bitcode holds the constants a module uses: the writer leaves this one out, with or without the order of uses, and the module db7fb25 parsed never had it.

  • A minimal source, in C++20:

    #include <source_location>
    
    unsigned line(std::source_location where = std::source_location::current())
    {
        return where.line();
    }
    
    unsigned here()
    {
        return line();
    }

    Compiled, .constant has the store and the unused struct as users. Read from the cache, it has the store.

  • loadsReadingStore does not meet them. It walks the users of the address a store writes to, and no store writes to .constant, a constant. The probe build counts 383,929 calls on the 53 sources and 21,904 on the 8 C++ files, none of them on a global variable. On Lua it counts 21,516 calls, 7 of them on a global variable, none of which has a constant among its users.

  • It can meet one in another source, and gives the same loads. Take a consteval function that returns a struct holding the address of a global variable gp, used as a default argument, and a function that writes gp. The walk meets the unused struct among the users of gp in the run that compiles, and not in the run that reads the cache. It keeps the users that are loads and asks MemorySSA about each, so a constant is passed over. Both runs give the same report, byte for byte.

  • The other walks: a walk that starts from a global variable or a function can meet such a constant, in the compiled module only. A mode of the probe build gives every global variable, then every defined function, of each loaded module an unused constant struct as a user. run_test.py passes 2760/2760 both ways, the reports of the 8 C++ files and of the 53 sources are the same byte for byte, and Lua's hold the same diagnostics and functions. Function::hasAddressTaken, which StackPointerEscapeResolver.cpp:115 asks, takes any user that is not a call for a taken address, such a constant included. No function has one on the three sets of sources.

  • Not done: removeDeadConstantUsers() on the globals of a compiled module would remove these constants, and the two modules would then list the same users. Nothing observed depends on them, so this pull request does not add it.

Reports from an empty cache and from a filled one

This branch at dbe9c03, --jobs=1, one compile IR cache directory per set of sources, empty before the first run:

Sources First run Second run Reports
8 C++ files 8 modules compiled 8 cache hits the same, byte for byte
This repository's 53 C++ sources 53 modules compiled 53 cache hits the same, byte for byte: 68 diagnostics, 3,304 functions
Lua 5.4.6, 34 C files 34 modules compiled 34 cache hits the same content: 1024 diagnostics and 2170 functions; the order of two diagnostics varies

For Lua, the content of the diagnostics is the same in both runs: compared without their ids and in a common order, the 1024 diagnostics are the same, and so are the 2170 functions. Their order can vary. Two CommandInjection.NonLiteralCommand diagnostics change places from one run to the next, with any build, and the ids follow the order. The cache is not what moves them: five runs that read the same filled cache give both orders. Lua's reports are compared by content, and are not the same byte for byte.

Not compared here: runs with more than one job. With #272, the comparison is to be made at one and four jobs, once coretrace-compiler holds both changes.

The test

testCompileIRCacheKeepsUseOrder, in the unit tests, takes the path of a run that fills the cache, then of a run that reads it:

  1. It compiles a C source of five lines through loadModuleForAnalysis, in the bitcode format, with a cache directory. This gives the module of the compilation, and an entry in the cache.
  2. It loads the source again, which gives the module parsed from that entry.
  3. It compares the two modules: the users of every global variable, function, argument, block and instruction, with the operand of each use, must be the same, in the same order.
  4. It writes the entry again from the compiled module, without the order of uses, and loads once more: the lists must now differ.

In the source, early calls late, defined after other, and the three call callee. Clang emits the bodies in the order of the source, while the module lists late, created by the call in early, before other. Bitcode follows the module's order, so a reader that is not told the order of uses lists the callers of callee another way.

Step 4 keeps the test from passing for nothing: it fails if the source stops telling the two orders apart, or if a load stops reading the entry. With ShouldPreserveUseListOrder taken out of InputPipeline.cpp, step 3 fails, and only it.

The test checks the order of uses on one C source. It does not compare reports: the comparisons above were run by hand.

A .ll given as input

Other code writes the cache entry of a .ll input, and this pull request does not change it. As on main, that entry's bitcode holds no order of uses, while the first run analyzes the module parsed from the text. For a .ll of three functions whose callee is declared after its two callers, as Clang prints declarations, the module parsed from the text and the module parsed from such bitcode list the callers in another order. The 12 .ll files of test/ show no such difference.

An existing test changes

check_compile_ir_format_switch recognized the bc format by the log line of the bitcode parse, which a compiled source no longer has. It changes in the commit that changes the log, and keeps its three checks, with one shared cache directory:

Run Before Now
bc, empty cache Bitcode parse done in is logged Compiled to a module in is logged, and neither parse line is
ll IR parse done in is logged, Bitcode parse done in is not the same, and Compiled to a module in is not logged either
bc again Bitcode parse done in is logged, IR parse done in is not Bitcode parse done in (the cache's bitcode) or Compiled to a module in (a build without the cache) is logged, IR parse done in is not

The changed test fails on db7fb25, where the first run logs Compilation done in, then Bitcode parse done in, and passes on this branch.

Measurements

Measured on 2026-11-12: coretrace-stack-analyzer db7fb25, with coretrace-compiler built from its main branch (f0a22d9), against the same analyzer with this change, on CoreTrace/coretrace-compiler#265. The measured build predates the log line of this branch and the fix of that pull request for LLVM 16, which LLVM 20 does not compile. macOS arm64 (M3, 8 cores), LLVM 20.1.2, --format=json --timing, no IR cache, an empty resource summary cache. Medians of 5 runs, 3 for the 53 sources; the builds alternate within each repetition.

Sources --jobs db7fb25 this change Loading: db7fb25, this change
Lua 5.4.6, 34 C files 1 7.70 s 7.72 s 1.14 s, 0.97 s
2 7.68 s 7.69 s 1.12 s, 0.97 s
8 C++ files, each including 11 standard headers 1 4.27 s 3.90 s 3.81 s, 3.49 s
2 4.39 s 4.02 s 4.03 s, 3.64 s
This repository's 53 C++ sources 1 55.0 s 51.9 s 44.1 s, 40.7 s
2 52.3 s 50.8 s 42.2 s, 40.3 s
  • Loading, at --jobs=1, takes 8 % less time on both sets of C++ sources and 15 % less on Lua's. Sources still compile one at a time here, whatever --jobs says: perf(input): compile sources in parallel, each in its directory #272 is the change that compiles them in parallel.
  • Reports: the same diagnostics and functions with both builds. The order of Lua's diagnostics, and so their ids, varies from run to run with either build.
  • Peak memory does not grow: the medians stay within the range of db7fb25's runs, or below it on Lua (166 MB against 177 to 189 MB at --jobs=1).

On Linux

Measured on 2026-11-17 on a GitHub runner (run 44158421994): ubuntu-24.04 on x86-64, 4 CPUs, g++ 13.3, LLVM 20, glibc 2.39, each run in a cgroup of 4 CPUs and 6 GiB without swap. main (588e800), which builds coretrace-compiler v0.10.0, against this branch (aa8b606), which builds v0.13.0. --jobs=1, --format=json --timing, no IR cache, an empty resource summary cache. Medians of 3 runs; the builds alternate from one repetition to the next. The branch is measured as it is, without main's last three pull requests (#283, #284, #285).

Sources main this branch
This repository's 53 C++ sources Whole run 118.8 s 118.1 s
Loading 108.2 s 106.1 s
Phases after loading 10.2 s 11.8 s
Peak resident memory 1009 MiB 1142 MiB
Lua 5.4.6, 34 C files Whole run 7.78 s 7.89 s
Loading 0.87 s 0.75 s
Phases after loading 6.88 s 7.10 s
Peak resident memory 169 MiB 173 MiB
  • The whole run takes the same time on the 53 sources: 117.8 to 118.8 s on main, 116.7 to 123.3 s on this branch.
  • Loading takes 2 % less time on the 53 sources, against 8 % in the macOS measure above, and 14 % less on Lua's.
  • The phases after loading take 16 % more time on the 53 sources and 3 % more on Lua's.
  • Peak memory grows by 13 % on the 53 sources.
  • These differences have two possible causes, which this measure does not tell apart: this change, and the compiler, v0.10.0 on main against v0.13.0 here. The macOS measure above builds both analyzers on the compiler's main branch at f0a22d9, the second with feat(compilerlib): compile to an llvm::Module in the caller's context coretrace-compiler#265 added.
  • Reports: the same diagnostics and functions with both builds, for each set of sources.
  • Another runner gives the same differences (run 44149941286), with a tree that holds main, this branch and perf(input): compile sources in parallel, each in its directory #272, at --jobs=1: 99.3 s on main against 100.6 s for the whole run, 90.6 s against 89.4 s for loading, 8.7 s against 11.2 s for the phases after loading, 1009 MiB against 1142 MiB.
  • At --jobs=2 and 4, the branch alone still loads on worker threads, which main stopped doing in fix(app): load the shared modules one at a time, on the calling thread #285: those runs say nothing of the merged result and are not shown.

Tests

Local runs, on macOS arm64 with AppleClang and LLVM 20.1.2, this branch built as a user builds it, coretrace-compiler fetched at the pinned commit, so that the compile IR cache is in use:

  • Functional tests, run_test.py --jobs 2 --no-cache, which runs the analyzer on the fixtures: 2760/2760 at dbe9c03, and at the three commits that change what is built, on main at 1fadba2: the pin alone (ae23d0b), the helper (3bbb15c) and the module path (02732dd). The script counts each unit test program below as one of the 2760, and as passed when the program is missing.
  • Unit tests, two programs that the script runs when they are built (-DBUILD_ANALYZER_UNIT_TESTS=ON). At dbe9c03 both were built and executed: stack_usage_analyzer_unit_tests ends with All analyzer module unit tests passed. after 399 checks, and ownership_engine_unit_tests with All ownership engine unit tests passed. after 116 checks. At ae23d0b, 3bbb15c and 02732dd they were not built here, so not executed. The CI builds both and requires them at every push.
  • The new unit test: its three checks pass at dbe9c03. With ShouldPreserveUseListOrder taken out, the module read from the cache lists each value's uses as the compiled module does fails, and the two other checks pass.
  • The changed test, run alone: it fails against db7fb25 built as published, with --compile-ir-format=bc did not take the module from the compilation, and passes at 02732dd. Its three runs then log Compiled to a module in, then Compilation done in and IR parse done in, then Compilation cache hit and Bitcode parse done in.
  • A filled cache against an empty one: the table of Reports from an empty cache and from a filled one.
  • scripts/format-check.sh with clang-format 20, scripts/check-license-compliance.sh and the commit checker pass.

CI at aa8b606

The head adds two commits to dbe9c03: the pin of v0.13.0 (6f50b2b) and its sentence in the changelog (aa8b606). They were not built locally: the runs below are their validation, on the merge of this branch into main at 588e800.

  • Build run 44154348044: the three jobs pass, on ubuntu-24.04, macos-15 and macos-latest. Each builds the unit tests and runs the regression tests: 2916/2916, and both unit test programs end with their All ... unit tests passed. line. The macos-latest job got no runner at first, and the run was started again.
  • Other runs: the release binaries for Linux amd64 and arm64, the CI integration (the GitHub Action, the Docker image, the consumer project), the concurrency analysis, clang-format, license-compliance and the commit conventions pass.

Merging

coretrace-compiler v0.13.0 brings the changes that this branch and #272 need, and this branch pins it. #272 also changes how InputPipeline.cpp runs a compilation: it is being reworked, and will have this branch merged in.

The pinned commit, from CoreTrace/coretrace-compiler#265, gives the
module of a compilation in the caller's context. The analyzer does not
use it yet.

Refs #273
…lper

The lock, the move to the compile command's directory and the timing
of a compilation now wrap a callable, which compile() is the only one
to use yet. No behaviour changes.

Refs #273
The bitcode format compiled each source to a temporary .bc file, read
it into a string, copied it into a buffer and parsed it. The module now
comes from compileToModule, created in the context of the load: nothing
is written to a temporary file, and nothing is parsed. With the compile
IR cache, the cache stores the bitcode of that module, written with the
order of each value's uses.

With --timing, a source compiled that way logs "Compiled to a module in
N ms", where it logged "Compilation done", then "Bitcode parse done".
check_compile_ir_format_switch recognized the bitcode format by the
parse line: it now expects the new line from an empty cache, and either
line once the cache is filled.

Refs #273
auto compileInWorkingDir =
[&](const std::vector<std::string>& compileArgs,
compilerlib::OutputMode outputMode) -> std::optional<compilerlib::CompileResult>
auto inWorkingDir = [&](auto&& compileHere) -> std::optional<compilerlib::CompileResult>
auto compileInWorkingDir =
[&](const std::vector<std::string>& compileArgs,
compilerlib::OutputMode outputMode) -> std::optional<compilerlib::CompileResult>
auto inWorkingDir = [&](auto&& compileHere) -> std::optional<compilerlib::CompileResult>
The analyses walk a module read from the compile IR cache as they
walked the compiled one only if the bitcode holds the order of each
value's uses: the first match of a walk and the order of a variable's
reports follow that order.

Refs #273
…f uses

A source compiled in the bitcode format is analyzed as the module of
its compilation, then read from the compile IR cache by the next run.
The test compiles a source whose calls Clang emits in another order
than the module lists their functions, and compares each value's users
between the two modules. A third load, on the entry written again
without the order of uses, gives another order: the source tells the
two apart, and a load reads the entry.

Refs #273
SizzleUnrlsd added a commit that referenced this pull request Nov 17, 2026
In the first measure, the phases after loading took longer in the rework
than on main, at --jobs=1 as well. The rework holds #274: a fourth
analyzer, the head of #274, tells its effect from that of the rework.

Refs #272
SizzleUnrlsd added a commit that referenced this pull request Nov 18, 2026
In the second measure, the phases after loading took longer with #274
alone than on main, and the peak of memory was higher. #274 also moves
coretrace-compiler from v0.10.0 to v0.13.0. Three analyzers tell the two
effects apart: main, main built with that compiler, and main merged with
#274. One value of --jobs is enough: main loads its inputs one after the
other, whatever --jobs.

Refs #274
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The bitcode format writes each module to a temporary file, then reads and parses it

2 participants