Skip to content

perf(input): compile sources in parallel, each in its directory - #272

Draft
SizzleUnrlsd wants to merge 7 commits into
mainfrom
perf/parallel-compilation
Draft

SizzleUnrlsd wants to merge 7 commits into
mainfrom
perf/parallel-compilation

Conversation

@SizzleUnrlsd

@SizzleUnrlsd SizzleUnrlsd commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Every compilation held gCompileInvokeMutex and moved the whole process to its compile command's directory, so the --jobs threads compiled one source at a time (#271). coretrace-compiler now gives each compilation the directory of its -working-directory without moving the process (CoreTrace/coretrace-compiler#262): the analyzer passes each compile command's directory that way, and compilations share the lock.

Closes #271

Change

  • build(deps): pin coretrace-compiler to its per-compilation working directory: a commit of feat(compilerlib): give each compilation a working directory of its own coretrace-compiler#262, a draft. This pull request waits for it, and should pin the release that brings it.
  • perf(input): compile sources in parallel, each in its directory:
    • -working-directory <directory> is prepended to the compile command's arguments; ScopedCurrentPath, which moved the process, is gone;
    • compilations take the lock shared. Those with an option that Clang applies to LLVM's state for the whole process (-mllvm, -mdebug-pass, -mlimit-float-precision, -ftime-report), or with -fno-integrated-cc1, whose jobs run as programs for which coretrace-compiler moves the process, take it alone, as coretrace-compiler asks.
  • docs(readme) and docs(changelog).
  • build(deps): pin coretrace-compiler to its fix of the files -MJ writes: a newer commit of feat(compilerlib): give each compilation a working directory of its own coretrace-compiler#262, which also writes the fragment of -MJ, the statistics of -save-stats, the graph of -dependency-dot and the module cache in the directory of -working-directory, where this analyzer wrote them when it moved the process there.
  • fix(input): bring the process back after a compilation that moves it, found in review, with its unit test. coretrace-compiler moves the process to the -working-directory of a compilation whose jobs run as programs, and leaves it there (feat(compilerlib): give each compilation a working directory of its own coretrace-compiler#262). ScopedCurrentPath brought it back; once it was gone, the process stayed in the compile command's directory after a load with -fno-integrated-cc1, and a later source given by a relative path, without a compile command, was not found. A compilation that takes the lock alone now brings the process back to the directory it was in, before it releases the lock. Compilations that share the lock do not move the process and are not changed. The command line does not reach the defect: given a compilation database, it refuses a source without a compile command. loadModuleForAnalysis does.

A compile command whose directory does not exist still fails the analysis, with the message it gave before. coretrace-compiler reports such a directory, but compiles to memory all the same (CoreTrace/coretrace-compiler#264), so the analyzer checks the directory itself before compiling.

Measurements

macOS arm64 (M3, 8 cores), LLVM 20.1.2, --format=json --timing, no IR cache, an empty resource summary cache. Both builds use coretrace-compiler from source: main (f0a22d9) for db7fb25, CoreTrace/coretrace-compiler#262 for this branch. Medians of 5 runs, 3 for the 53 sources; the builds alternate within each repetition.

Sources --jobs db7fb25 this branch Loading: db7fb25, this branch
Lua 5.4.6, 34 C files 1 7.70 s 7.71 s 1.14 s, 1.17 s
2 7.68 s 7.51 s 1.12 s, 0.90 s
8 C++ files, each including 11 standard headers 1 4.27 s 4.07 s 3.81 s, 3.68 s
2 4.39 s 2.61 s 4.03 s, 2.22 s
This repository's 53 C++ sources 1 55.0 s 55.1 s 44.1 s, 43.9 s
2 52.3 s 34.1 s 42.2 s, 23.7 s
4 53.0 s 25.3 s 43.2 s, 15.2 s
  • The --jobs=4 row was measured apart, on 2026-11-15, on the 53 sources only: this branch at e6ae53e, with coretrace-compiler fetched at the pinned commit, against db7fb25 built as above. Each of the 6 runs started once the machine's processors had been at least 80 % idle for two samples in a row, and the three runs of each build differ by less than 0.3 s.
  • Loading with 4 jobs takes 2.8 times less, not 4: the 53 compilations add up to 53.4 s, where they add up to 42.8 s one at a time.
  • Reports: the same diagnostics and functions with both builds, at every job count measured. The order of Lua's diagnostics, and so their ids, varies from run to run with either build.
  • Peak memory grows with the compilations in flight: at --jobs=2, 239 MB to 328 MB for the 8 C++ files, 1.26 GB to 1.42 GB for the 53 sources; at --jobs=4, 1.35 GB to 1.62 GB for the 53 sources.
  • The analysis is not changed: on Lua it takes 6.5 s of the 7.7 s (Multi-file runs analyze their modules one at a time, whatever --jobs #199), and on the 53 sources at --jobs=4, 9.7 s of the 25.3 s.
  • More than 4 jobs, and 4 jobs on the other two sets of sources, were not measured, to keep the machine usable.

Tests

Local runs, on macOS arm64 with AppleClang and LLVM 20.1.2, this branch built as a user builds it, coretrace-compiler fetched at the pinned commit, so that the compile IR cache is in use:

  • run_test.py --jobs 2 --no-cache: 2696/2696 at the head, e6ae53e, at d642039, before the second pin, and at the first commit, the first pin alone. check_resource_model_across_compile_directories compiles eight sources, each from a directory of its own with a header of the same name as a decoy in the start directory, at the default, 1 and 4 jobs.
  • A compile command whose directory does not exist: the analysis fails with Failed to change working directory to: <directory>, as with db7fb25.
  • The review fix, at bbaab08, built with -DBUILD_ANALYZER_UNIT_TESTS=ON:
    • unit tests: stack_usage_analyzer_unit_tests passes its 388 checks and ownership_engine_unit_tests its 93. The new test loads project/unit.c, whose compile command has -fno-integrated-cc1, from the directory above, checks that the process is still there, then loads host.c by its relative path. At e6ae53e, the last two of its three checks fail;
    • functional tests: run_test.py --jobs 2 --no-cache, 2696/2696.
  • scripts/format-check.sh with clang-format 20, scripts/check-license-compliance.sh and the commit checker pass.

CI at bbaab08

  • CI run 43786937373: the three jobs pass, on ubuntu-24.04, macos-15 and macos-latest.
    • unit tests: both programs run and pass in each job, and so do the three checks of the new test;
    • functional tests: 2760/2760 in each job.
  • Other checks: the Linux binaries (amd64 and arm64), the Docker image, the GitHub Action, the consumer project, the concurrency analysis, format-check, license-compliance and the commit conventions pass.

…rectory

The pinned commit, from CoreTrace/coretrace-compiler#262, gives each
compilation a file system of its own, whose working directory
-working-directory moves while the process's stays. The analyzer does
not use it yet: it still moves the process around each compilation,
under its lock.

Refs #271
Every compilation held one global lock and moved the whole process to
its compile command's directory: the --jobs threads compiled one
source at a time. coretrace-compiler now gives each compilation the
directory of its -working-directory without moving the process: the
analyzer passes the directory that way, and compilations share the
lock.

Some compilations still hold it alone, since they change state that
the whole process shares: those with an option that Clang applies to
LLVM's state (-mllvm, -mdebug-pass, -mlimit-float-precision,
-ftime-report), and those with -fno-integrated-cc1, whose jobs run as
programs, for which coretrace-compiler moves the process.

A compile command whose directory does not exist still fails the
analysis: coretrace-compiler reports a directory it cannot enter, but
compiles to memory all the same, so the analyzer checks the directory
itself, with the message it gave before.

Refs #271
@SizzleUnrlsd SizzleUnrlsd added the enhancement New feature or request label Nov 12, 2026
@SizzleUnrlsd SizzleUnrlsd self-assigned this Nov 12, 2026
The newer commit of CoreTrace/coretrace-compiler#262 also writes the
fragment of -MJ, the statistics of -save-stats, the graph of
-dependency-dot and the module cache in the directory of
-working-directory, as the analyzer did when it moved the process
there.

Refs #271
With -fno-integrated-cc1, the jobs of a compile command run as
programs: coretrace-compiler moves the process to the command's
directory and leaves it there. The scoped change of directory that
this branch removed used to bring the process back. After a load of
such a source, the process stayed in the command's directory, where a
relative source without a compile command was no longer found. A
compilation that holds the lock alone now puts the process back before
it releases the lock. The compilations that share the lock do not move
the process, and still run at once.

Refs #271
A source whose compile command passes -fno-integrated-cc1 is loaded,
then a relative source without a compile command. Before the fix, the
process was left in the command's directory and the second load
failed.

Refs #271
SizzleUnrlsd added a commit that referenced this pull request Nov 17, 2026
The table of each phase was kept for the first repetition only: the
phases after loading can now be compared over every repetition.

Refs #272
SizzleUnrlsd added a commit that referenced this pull request Nov 17, 2026
In the first measure, the phases after loading took longer in the rework
than on main, at --jobs=1 as well. The rework holds #274: a fourth
analyzer, the head of #274, tells its effect from that of the rework.

Refs #272
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sources compile one at a time, whatever --jobs

1 participant