Skip to content

feat(compilerlib): compile to an llvm::Module in the caller's context - #265

Merged
SizzleUnrlsd merged 28 commits into
mainfrom
feat/compile-to-module
Nov 16, 2026
Merged

SizzleUnrlsd merged 28 commits into
mainfrom
feat/compile-to-module

Conversation

@SizzleUnrlsd

@SizzleUnrlsd SizzleUnrlsd commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The CoreTrace analyzers compile each source to textual IR or bitcode, then parse it into a module of their own (#263). compileToModule(args, context, instrument), in a new public header compilerlib/compile_module.hpp, gives the module itself, created in an llvm::LLVMContext of the host. The output file is not written; the other files the arguments ask for, such as a dependency file, are.

Closes #263

Change

  • refactor(compilerlib): run compile() through runCompilation: the compilation runs from the context it is given. compile does not change.
  • feat(compilerlib): compile to an llvm::Module in the caller's context:
    • compileToModule takes the arguments of an LLVM IR compilation, -S -emit-llvm or -c -emit-llvm, and fails on any other;
    • code generation creates the module in the host's context, through the action that compile already uses for in-memory output; with instrument, the module is instrumented first;
    • whether the context discards value names is restored after the compilation, since it applies to the modules read into the context later. Before LLVM 22, the optimization record that Clang's action leaves on the context is removed too.
  • test(compilerlib): check the modules compileToModule gives: the module prints as the text compile gives, at -O0 and -O2, instrumented or not; a bitcode compilation writes its dependency file and no output; a compilation to an object and a broken source fail; the context keeps its setting of value names.
  • docs(readme): describe compiling to a module.
  • fix(compilerlib): keep value names in an LLVM 16 context given some, with its README and changelog notes. The Linux / LLVM 16 job of this pull request aborted in ModuleLeavesTheValueNamesSettingOfTheContext: double free or corruption. With LLVM 16, Value::setName("") does nothing in a context that discards names, so destroying a module whose values have names, after the context was set back to discarding them, left each argument's name in its function's symbol table, which freed it again. LLVM 17 removes the name (16.0.6 against 17.0.6, Value::setNameImpl). With LLVM 16, a context that discards names now keeps them after a compilation that names values; the test expects that with LLVM 16, and the restored setting from LLVM 17 on.
  • fix(compilerlib): give a module's context its diagnostic handler back, found in review, with its header, README and changelog notes. Clang's backend replaces the context's diagnostic handler with one that refers to the compilation, and returns without putting the previous one back when it cannot set up the optimization record of -fsave-optimization-record, or cannot link in a module of -mlink-bitcode-file (BackendConsumer::HandleTranslationUnit; 16.0.6, 18.1.8 and 22.1.0 do the same). The host's context kept the handler of a compilation that had ended, and the next diagnostic the host emitted crashed: a segmentation fault on macOS arm64. compileToModule now sets the host's handler aside before the job and puts it back once the job ends, however it ends. The job runs with LLVM's default handler, as a compilation in a context of its own does.
  • fix(compilerlib): restore a module's context after a recovered crash: the scope that restores the context is on the frame the job returns to, whatever happens to the job. After a crash that the host recovered, the context kept the job's setting of value names.
  • test(compilerlib), six tests of the host's handler: it is the context's, and is called for the host's next diagnostic, after a compilation that succeeds; after an optimization record file that cannot be opened, an unknown record format, an instrumented compilation that fails and a module that cannot be linked in, each followed by a compilation that succeeds in the same context; and after a crash that the host recovers, where the context has its setting of value names too.

Two limits of the handler fix:

  • During the compilation, the host's handler is not called. A diagnostic that LLVM reports before Clang's backend installs its own handler gets LLVM's default handling. None was found in a C or C++ compilation.
  • LLVMContext::setDiagnosticHandler also resets whether the context respects diagnostic filters, which LLVM gives no way to read. A host that had set it finds it reset, as it already did after a compilation that succeeded, since Clang's backend puts the handler back the same way.

This branch comes from commit 2c1c83b of the experimental branch, without what it took from the working directory change, which #262 brought separately.

Integration of #262. #262 is merged, and main (e6d5c6c) is merged into this branch (be9bd28). The two changed compile in compiler.cpp:

  • src/compilerlib/compiler.cpp: this branch runs compile() through runCompilation(). What feat(compilerlib): give each compilation a working directory of its own #262 changed in the body of compile(), the move of the process for a compilation with a job that runs as a program, is applied there. compileToModule creates its compiler instance as compile does for an output in memory, so it reads and writes through the file system of its compilation.
  • test/unit/compile_test.cpp, README.md, CHANGELOG.md: both sides are kept.
  • test(compilerlib): check modules compiled from several threads at once: compileToModule from eight threads, each with a context and a project of its own, whose headers have the same name. Each module comes from its own header, and the process's working directory stays where it is, instrumented or not. The test fails when the compilations share the process's file system, as they did before feat(compilerlib): give each compilation a working directory of its own #262.
  • docs: the header, the README and the changelog say that -working-directory and compilations from several threads are as for compile, each compilation with a context of its own.

A compilation that compileToModule refuses for its jobs, with -working-directory, leaves the process in that directory, as compile does for an output in memory, and as main did before #262: #267.

Measurements

coretrace-stack-analyzer db7fb25, with coretrace-compiler built from main (f0a22d9), against the same analyzer taking each module from compileToModule in its bitcode mode, instead of compiling to a temporary .bc file, reading it and parsing it. That analyzer change is CoreTrace/coretrace-stack-analyzer#274, a draft; the measured build predates the log line it adds and the LLVM 16 fix of this branch, which LLVM 20 does not compile. macOS arm64 (M3, 8 cores), LLVM 20.1.2, --format=json --timing, no IR cache, an empty resource summary cache. Medians of 5 runs, 3 for the 53 sources; the builds alternate within each repetition.

Sources --jobs main this branch Loading: main, this branch
Lua 5.4.6, 34 C files 1 7.70 s 7.72 s 1.14 s, 0.97 s
2 7.68 s 7.69 s 1.12 s, 0.97 s
8 C++ files, each including 11 standard headers 1 4.27 s 3.90 s 3.81 s, 3.49 s
2 4.39 s 4.02 s 4.03 s, 3.64 s
coretrace-stack-analyzer's 53 C++ sources 1 55.0 s 51.9 s 44.1 s, 40.7 s
2 52.3 s 50.8 s 42.2 s, 40.3 s
  • Loading, at --jobs=1, takes 8 % less time on both sets of C++ sources and 15 % less on Lua's. The analyzer still compiles one source at a time here, whatever --jobs says; feat(compilerlib): give each compilation a working directory of its own #262 is the change that lets it compile in parallel.
  • Reports: the same diagnostics and functions with both builds. The order of Lua's diagnostics, and so their ids, varies from run to run with either build.
  • Peak memory does not grow: the medians stay within the range of main's runs, or below it on Lua (166 MB against 177 to 189 MB at --jobs=1).

Tests

Local runs, on macOS arm64 with AppleClang:

Run Result
LLVM 20.1.2, ctest -j2 265/265, the 257 of main plus 8 new tests.
LLVM 16.0.6, Debug, before the LLVM 16 fix ModuleLeavesTheValueNamesSettingOfTheContext aborts: pointer being freed was not allocated, in ~ValueSymbolTable from ~Module, as on the CI's Linux / LLVM 16 job.
LLVM 16.0.6, after the fix all 171 unit tests pass.
LLVM 20.1.2, the six tests of the handler before the two review fixes 3 fail where they check the handler: the record file, the record format, the module to link in. The test of the recovered crash exits with status 2, the setting of value names. The other two pass.
LLVM 20.1.2, at 9ebd2dd, ctest 271/271: the 265 above plus the six tests of the handler. The 177 unit tests of compilerlib also pass in a shuffled order.
LLVM 16.0.6, at 9ebd2dd all 177 unit tests pass.
LLVM 20.1.2, at be9bd28, the merge of main alone ctest -j2 307/307; the 213 unit tests of compilerlib pass in a shuffled order.
LLVM 16.0.6, at be9bd28 all 213 unit tests pass.
LLVM 20.1.2, at 6eb4646 ctest -j2 309/309: the 293 of main plus 16 tests of this branch. The 215 unit tests of compilerlib pass in a shuffled order, and the new test five times in a row.
LLVM 16.0.6, at 6eb4646 all 215 unit tests pass.

Each commit builds and passes the unit tests with LLVM 20; of the 12 commits of the review fixes, the 9 that change code or tests were each built and tested. With LLVM 16, ModuleLeavesTheValueNamesSettingOfTheContext fails from the commit that adds it until the fix. clang-format 18, scripts/check-license-compliance.sh and the commit checker pass.

CI at 6eb4646

compile() built its context and ran the whole compilation itself. The
compilation now runs in runCompilation, from the context it is given,
which compileToModule will share.

Refs #263
The analyzers compile each source to textual IR or bitcode, then read
it back into a module of their own: the module was written and parsed
again for nothing. compileToModule(args, context, instrument) gives
the module itself, created in the caller's context, with the arguments
of an LLVM IR compilation (-S -emit-llvm or -c -emit-llvm). The output
file is not written; the other files the arguments ask for, such as a
dependency file, are.

The module is the one compile() writes, instrumented or not. Code
generation sets whether the context discards value names; the
context's own setting is restored, since it applies to the modules
read into it later, and so is the optimization record that Clang's
action leaves on it before LLVM 22.

Refs #263
compileToModule gives the module that compile() writes as text, at -O0
and -O2, instrumented or not, and the module of a bitcode compilation,
whose output file is not written and whose dependency file is. It
fails on a compilation that is not to LLVM IR, reports the errors of a
broken source, and leaves the context's setting of value names.

Refs #263
@SizzleUnrlsd SizzleUnrlsd added the enhancement New feature or request label Nov 12, 2026
@SizzleUnrlsd SizzleUnrlsd self-assigned this Nov 12, 2026
With LLVM 16, destroying a module of compileToModule freed memory twice
when the compilation named values and the context, restored after it,
discarded names: LLVM 16's Value::setName("") does nothing in such a
context, so Function::clearArguments leaves each argument's name in
the function's symbol table, which frees it again. LLVM 17 removes the
name. With LLVM 16, a context that discards names now keeps them after
a compilation that names values. The Linux / LLVM 16 job of the CI
aborted in ModuleLeavesTheValueNamesSettingOfTheContext.

Refs #263
compileToModule restored the host's context from the job itself. Once
the host enabled crash recovery, a crash leaves the job where it stood:
the context kept the job's setting of value names. The context is now
restored in runToModule, which the job always returns to.

Refs #263
Clang's backend replaces the context's diagnostic handler with one that
refers to the compilation, and returns without putting the previous one
back when it cannot set up the optimization record or link a module in.
After such a compileToModule, the host's handler was destroyed, and the
next diagnostic of the context went to the compilation that had ended:
a segmentation fault. The host's handler is now set aside for the job,
which runs with LLVM's default one, and put back when the job ends.

Refs #263
The compilation fails on an optimization record whose directory does
not exist. The context must still have the host's handler, report to
it, and take another compilation. Before the fix, the context was left
with the handler of Clang's backend.

Refs #263
Instrumented, compilerlib sets up the optimization record itself. This
path already put the handler back.

Refs #263
The other early return of Clang's backend: a module given with
-mlink-bitcode-file defines the unit's symbols again.

Refs #263
The source crashes Clang once the host enabled crash recovery. The
context must keep the host's setting of value names and its diagnostic
handler. Before the fix, it kept the job's setting of value names.

Refs #263
main brings the working directory of each compilation (#262) and the
note about an allocation in a do-while loop (#260).

Conflicts:
- src/compilerlib/compiler.cpp: this branch runs compile() through
  runCompilation(). What #262 changed in the body of compile(), the move
  of the process for a compilation with a job that runs as a program,
  is applied there.
- test/unit/compile_test.cpp: both includes are kept.
- README.md and CHANGELOG.md: both sections and both entries are kept.
compileToModule from eight threads, each with a context and a project
of its own, whose headers have the same name: each module comes from its
own header, and the process's working directory stays where it is,
instrumented or not. The test fails when the compilations share the
process's file system, as they did before #262.

Refs #263
@SizzleUnrlsd
SizzleUnrlsd marked this pull request as ready for review November 16, 2026 15:30
@SizzleUnrlsd
SizzleUnrlsd merged commit 61f88b5 into main Nov 16, 2026
49 checks passed
@SizzleUnrlsd
SizzleUnrlsd deleted the feat/compile-to-module branch November 16, 2026 16:40
SizzleUnrlsd added a commit to CoreTrace/coretrace-stack-analyzer that referenced this pull request Nov 17, 2026
The pinned commit was one of the branch of
CoreTrace/coretrace-compiler#265. v0.13.0 is the release that holds
compileToModule.

Refs #273
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hosts get each module as text or bitcode, then parse it back

1 participant