Skip to content

fix(app): load the shared modules one at a time, on the calling thread - #285

Merged
SizzleUnrlsd merged 4 commits into
mainfrom
fix/sequential-shared-loading
Nov 16, 2026
Merged

SizzleUnrlsd merged 4 commits into
mainfrom
fix/sequential-shared-loading

Conversation

@SizzleUnrlsd

@SizzleUnrlsd SizzleUnrlsd commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A run of more than one input loads its inputs together (analyzeWithSharedModuleLoading), for the cross-TU summaries. With --jobs above 1, as by default (auto, the number of hardware threads), it loaded them on min(jobs, inputs) workers. Compilations run one at a time under gCompileInvokeMutex (InputPipeline.cpp), so these workers only read bitcode in parallel. Each worker allocates before it waits for that lock, so glibc gives each one an arena of its own. Each arena keeps the memory its compilation freed, and no other compilation runs there afterwards. glibc may hand these arenas to the cross-TU summaries' workers, but most of that memory stays free: the module loop allocates in the main arena, and at the end of the run the arenas still keep about one compilation's worth of free memory per worker.

This pull request loads the inputs one after the other, on the calling thread, whatever --jobs. The cross-TU summaries keep their workers. By default, this changes every run of more than one input on a machine with several hardware threads.

Merge order: after #284, which is merged. Alone, this change makes the order of a function's GlobalReadBeforeWrite issues vary from run to run at --jobs above 1, as it already did at --jobs=1 (see Outputs); #284 emits them in the order of the instructions. This branch was then updated with main (280ba46), and CI ran the two together (see CI and code scanning).

It was found while validating #239: four analyzers at once, each given 14 of this repository's sources at --jobs=32 with #239's check on, ran out of a 10 GiB container.

Change

  • fix(app): load the shared modules one at a time, on the calling thread: in src/app/AnalyzerApp.cpp, the if/else that chose between a loop and runParallelWork becomes the loop, with a comment that says why. Two sentences of README.md on --jobs change with it.
  • docs(readme): say that --jobs parallelizes only the summaries of multi-file runs: the sentence the first commit wrote said that --jobs parallelized loading and analysis in multi-file runs without cross-TU. There are no such runs: every run of more than one input takes the shared loading, for the global read-before-write summaries, and analyzes its modules one after the other.
  • docs: describe --jobs as it works in --help and the README's option list: both still said that --jobs parallelizes multi-file loading and analysis, with a default of 1; it defaults to auto. The README's sentence also no longer gives the serialized compilations as the reason for loading one input at a time, a reason that does not hold for an input already in LLVM IR or found in the compile IR cache.

Kept: the modules stay in the order of the inputs; every input is loaded before a failure is reported, the first in the order of the inputs; the compile lock and the handling of the working directory do not change. A run of one input does not take this path.

Diagnosis

On #239's head (57aa98a), whose loading code and compile lock are the same as on main at 1fadba2. Linux x86_64 with glibc, a container of 4 CPUs and 10 GiB, one analyzer at a time, #239's check off, four groups of 14 of this repository's sources. A probe preloaded into the analyzer sampled mallinfo2() every 250 ms and malloc_info() every second. --jobs=32, which creates 14 loading workers, against --jobs=1:

  • Not memory in use: at the end of loading, the bytes in use in glibc's arenas change by −38 to +147 MiB, while the free bytes that the allocator keeps grow by 1729 to 2347 MiB, and RssAnon by 1650 to 2445 MiB.
  • In the workers' arenas: 15 arenas instead of 1. The 14 workers' arenas got 2189 to 3315 MiB from the system, of which 2021 to 2993 MiB are free; the main arena holds 7 to 17 MiB.
  • Left by the compilations: after each compilation on a worker, the free memory kept grows by a median step of 157 to 182 MiB, and does not come down. On one thread, each compilation reuses what the previous one freed (median step −5 to +6 MiB).
  • Kept to the end of the run: after the module loop, which runs on one thread, the arenas still keep 1999 to 3311 MiB free.
  • With MALLOC_ARENA_MAX=1, RssAnon after loading comes back within 7 MiB of the --jobs=1 level.
  • The workers gain nothing: at --jobs=1, the compilations under the lock take 21.0 to 32.9 s of 21.5 to 34.0 s of loading. Reading the bitcode, the only part the workers run in parallel, takes 0.5 to 1.1 s in all. With 14 workers, loading took 2.9 to 5.5 s longer.

Set aside:

  • MALLOC_ARENA_MAX or mallopt: a process-wide glibc policy, and each run was 0.5 to 8.7 s longer than at --jobs=1 with it;
  • malloc_trim after loading: glibc-specific, and it would return the memory after the fact, while it accumulates during loading;
  • a compile thread shared by every path: it would keep the parallel reading of bitcode, at the cost of a queue and a handoff.

Verification

Release builds of 57aa98a and of 57aa98a with the first commit, from one tree, on a second Linux x86_64 machine, in a container of 4 CPUs and 5 GiB without swap. The same four groups of sources, one run per cell, without and with #239's check, the runs of the two builds interleaved.

Memory

#239's check off; MiB; "before / after":

Group --jobs Threads while loading RssAnon at the end of loading Peak RSS
0 1 1 / 1 440 / 440 546 / 544
0 4 5 / 1 974 / 438 1101 / 590
0 32 15 / 1 2222 / 439 2364 / 613
1 1 1 / 1 840 / 840 981 / 981
1 4 5 / 1 1822 / 838 2025 / 1047
1 32 15 / 1 3280 / 840 3495 / 1090
2 1 1 / 1 374 / 373 483 / 482
2 4 5 / 1 977 / 374 1140 / 582
2 32 15 / 1 2418 / 386 2579 / 607
3 1 1 / 1 509 / 509 610 / 609
3 4 5 / 1 1119 / 509 1261 / 700
3 32 15 / 1 2155 / 510 2319 / 741
  • At --jobs=1, both builds take the same path: −12 to +8 MiB, the order of the noise of these samples.
  • At --jobs=32, without and with feat(const-param): experimental Clang check of const suggestions, version 2 #239's check, the change removes 1644 to 2451 MiB of RssAnon at the end of loading, which comes back within 13 MiB of its --jobs=1 level, and 1579 to 2793 MiB of peak RSS. At --jobs=4, it removes 536 to 984 MiB at the end of loading.
  • What stays above --jobs=1 at --jobs=32 (+69 to +161 MiB at the peak) comes after loading, from the cross-TU summaries, which keep their workers.

Time

Seconds, one run each. The pairs at --jobs=1 give the noise: −1.7 to +1.1 s per run.

  • Loading at --jobs=32: 1.0 to 3.8 s shorter. Before, it took 2.4 to 5.4 s more than at --jobs=1; after, it stays within −0.2 to +2.0 s of its own --jobs=1.
  • Whole runs at --jobs=32: 1.9 to 9.2 s shorter with feat(const-param): experimental Clang check of const suggestions, version 2 #239's check off, 2.7 to 39.0 s with it on, and 2.3 to 4.7 s less system time. At --jobs=4: −0.9 to +0.1 s, within the noise.
  • feat(const-param): experimental Clang check of const suggestions, version 2 #239's check's step, before the change, ran slower at --jobs=32 than at --jobs=1 on this machine: +0.9 to +33.1 s of user CPU time, without page faults or memory pressure. After the change, it stays within −0.8 to +0.3 s of its --jobs=1 time. The cause is not established; on the first machine, that effect was +0.4 to +1.4 s.

These are memory measurements; the times are given for completeness, not as a speedup.

Outputs

Tests

On 57aa98a with the first commit: CI's build passes, with one warning that was already there, in a unit test file this change does not touch. The unit and ownership tests pass, and the suite passes 2742/2742 as CI runs it (#239's suite, its own checks included). It already compares outputs at --jobs 1, 2 and 4.

Four analyzers at once

With the change, #239's check on, and the automatic --jobs (12 on that machine), all four finished. The container peaked at 3.97 of 5 GiB, and the analyzers' summed RssAnon at 3.20 GiB. The build without the change was not run in that configuration.

Interaction with #272

#272 (draft) lets compilations run in parallel. This change rests on compilations being serialized, as they are on main at 1fadba2. The two conflict:

#272 stays separate: before it lands, it will have to keep this memory gain and control the concurrency of its compilations. Nothing here measures #272's branch.

CI and code scanning

  • CI on 280ba46, this branch updated with main at b7afcfb, which holds fix(analysis): emit global read-before-write issues in instruction order #284 (merge ref b3bfc2a): every check passes; the release job is skipped, as outside tags.
  • The suite passes 2916/2916 on ubuntu-24.04, macos-15 and macos-latest, as main at b7afcfb does. fix(analysis): emit global read-before-write issues in instruction order #284's order check passes there at --jobs=1 and at --jobs=4 on the three runners, with the inputs now loaded one at a time.
  • What this CI does not measure: memory and time. The figures above come from feat(const-param): experimental Clang check of const suggestions, version 2 #239's head.
  • Code scanning on b3bfc2a: coretrace-stack-analyzer reported 72 results, one fewer than on main at b7afcfb, and coretrace-concurrency-analyzer none. New alerts: none.
    • The result fewer is a ConstParameterNotModified.Pointer note at src/app/AnalyzerApp.cpp:246, in the lambda of runParallelWork. main reports it once per instantiation of runParallelWork, five times; this change removes the instantiation for loading. The other results are those of main, six of them five lines higher in AnalyzerApp.cpp.
    • Of the 66 alerts open on main, 65 are open on refs/pull/285/merge. The one that is not, alert 156, is one of the five at that line.
  • Earlier:
    • on e60a10e (merge ref 2d666b2, into main at 9393ef3): every check passed; the suite passed 2915/2915 on the three runners, as main did then; code scanning showed the same difference;
    • on 4ae52d7 (merge ref c6d7220, into main at 1fadba2): every check passed, and the suite passed 2760/2760 on the three runners, as main did then.

Reservations

  • Measured on feat(const-param): experimental Clang check of const suggestions, version 2 #239's head, not on main. The loading code and the compile lock are the same on main at 1fadba2, but memory and time were not measured there. This pull request's CI checks the integration.
  • PCRE2 took 67.6 s longer with the change (+2.8 %): 2463.4 against 2395.8 s, one run each, two analyzers at once. The noise of this corpus is not measured; nothing is concluded.
  • Loads without a compilation are not measured. A hit of the compile IR cache (--compile-ir-cache-dir, which the suite uses in CI) or an input already in LLVM IR (.ll, .bc) does not take the compile lock. Before, such inputs were read in parallel; now they are read one after the other. Their time and memory are not measured.
  • glibc only. The arena behaviour is read from glibc's code and documentation, and measured on Linux; macOS's allocator was not measured.
  • One run per cell, on one fixed partition: the values per group hold for it only.

With cross-TU summaries, every input is loaded before the analysis. With
--jobs above 1, each input was loaded on a worker thread, yet its
compilation runs under gCompileInvokeMutex: a worker only parsed its
bitcode in parallel. glibc gives each of those workers an arena of its
own, and the memory a compilation frees stays in that arena after the
worker ends, for the rest of the run. On the self-analysis's chunks of 14
files, an analyzer at --jobs=32 held 1.6 to 2.4 GiB more after loading
than at --jobs=1, almost all of it free memory kept by the 14 worker
arenas, and its loading was no faster.

Load the inputs one after the other on the calling thread, whatever
--jobs. The cross-TU summaries still run on --jobs workers. The README no
longer says that --jobs parallelizes the loading of cross-TU runs.
…i-file runs

A run of more than one input always takes the shared loading, as it builds the global
read-before-write summaries, and it analyzes its modules one after the other. With the
previous commit, it also loads them one after the other: --jobs only parallelizes the
construction of the cross-TU summaries. The sentence the previous commit wrote said that
--jobs parallelized loading and analysis in multi-file runs without cross-TU.
--help and the README's list of options still said that --jobs parallelizes the loading
and the analysis of multi-file runs, with a default of 1. A run of more than one input loads
and analyzes its modules one after the other, and --jobs only parallelizes the construction
of its cross-TU summaries; it defaults to auto, the number of hardware threads
(resolveConfiguredJobs). The sentence the previous commit wrote also gave the serialized
compilations as the reason for loading one input at a time, which does not hold for an input
already in LLVM IR or found in the compile IR cache: it now only says what happens.
@SizzleUnrlsd
SizzleUnrlsd marked this pull request as ready for review November 16, 2026 04:18
@SizzleUnrlsd
SizzleUnrlsd merged commit 588e800 into main Nov 16, 2026
15 checks passed
@SizzleUnrlsd
SizzleUnrlsd deleted the fix/sequential-shared-loading branch November 16, 2026 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant