Repository navigation
fix(app): load the shared modules one at a time, on the calling thread - #285
Merged
Merged
Conversation
With cross-TU summaries, every input is loaded before the analysis. With --jobs above 1, each input was loaded on a worker thread, yet its compilation runs under gCompileInvokeMutex: a worker only parsed its bitcode in parallel. glibc gives each of those workers an arena of its own, and the memory a compilation frees stays in that arena after the worker ends, for the rest of the run. On the self-analysis's chunks of 14 files, an analyzer at --jobs=32 held 1.6 to 2.4 GiB more after loading than at --jobs=1, almost all of it free memory kept by the 14 worker arenas, and its loading was no faster. Load the inputs one after the other on the calling thread, whatever --jobs. The cross-TU summaries still run on --jobs workers. The README no longer says that --jobs parallelizes the loading of cross-TU runs.
…i-file runs A run of more than one input always takes the shared loading, as it builds the global read-before-write summaries, and it analyzes its modules one after the other. With the previous commit, it also loads them one after the other: --jobs only parallelizes the construction of the cross-TU summaries. The sentence the previous commit wrote said that --jobs parallelized loading and analysis in multi-file runs without cross-TU.
--help and the README's list of options still said that --jobs parallelizes the loading and the analysis of multi-file runs, with a default of 1. A run of more than one input loads and analyzes its modules one after the other, and --jobs only parallelizes the construction of its cross-TU summaries; it defaults to auto, the number of hardware threads (resolveConfiguredJobs). The sentence the previous commit wrote also gave the serialized compilations as the reason for loading one input at a time, which does not hold for an input already in LLVM IR or found in the compile IR cache: it now only says what happens.
SizzleUnrlsd
marked this pull request as ready for review
November 16, 2026 04:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A run of more than one input loads its inputs together (
analyzeWithSharedModuleLoading), for the cross-TU summaries. With--jobsabove 1, as by default (auto, the number of hardware threads), it loaded them onmin(jobs, inputs)workers. Compilations run one at a time undergCompileInvokeMutex(InputPipeline.cpp), so these workers only read bitcode in parallel. Each worker allocates before it waits for that lock, so glibc gives each one an arena of its own. Each arena keeps the memory its compilation freed, and no other compilation runs there afterwards. glibc may hand these arenas to the cross-TU summaries' workers, but most of that memory stays free: the module loop allocates in the main arena, and at the end of the run the arenas still keep about one compilation's worth of free memory per worker.This pull request loads the inputs one after the other, on the calling thread, whatever
--jobs. The cross-TU summaries keep their workers. By default, this changes every run of more than one input on a machine with several hardware threads.Merge order: after #284, which is merged. Alone, this change makes the order of a function's
GlobalReadBeforeWriteissues vary from run to run at--jobsabove 1, as it already did at--jobs=1(see Outputs); #284 emits them in the order of the instructions. This branch was then updated withmain(280ba46), and CI ran the two together (see CI and code scanning).It was found while validating #239: four analyzers at once, each given 14 of this repository's sources at
--jobs=32with #239's check on, ran out of a 10 GiB container.Change
fix(app): load the shared modules one at a time, on the calling thread: insrc/app/AnalyzerApp.cpp, theif/elsethat chose between a loop andrunParallelWorkbecomes the loop, with a comment that says why. Two sentences ofREADME.mdon--jobschange with it.docs(readme): say that --jobs parallelizes only the summaries of multi-file runs: the sentence the first commit wrote said that--jobsparallelized loading and analysis in multi-file runs without cross-TU. There are no such runs: every run of more than one input takes the shared loading, for the global read-before-write summaries, and analyzes its modules one after the other.docs: describe --jobs as it works in --help and the README's option list: both still said that--jobsparallelizes multi-file loading and analysis, with a default of 1; it defaults toauto. The README's sentence also no longer gives the serialized compilations as the reason for loading one input at a time, a reason that does not hold for an input already in LLVM IR or found in the compile IR cache.Kept: the modules stay in the order of the inputs; every input is loaded before a failure is reported, the first in the order of the inputs; the compile lock and the handling of the working directory do not change. A run of one input does not take this path.
Diagnosis
On #239's head (
57aa98a), whose loading code and compile lock are the same as onmainat1fadba2. Linux x86_64 with glibc, a container of 4 CPUs and 10 GiB, one analyzer at a time, #239's check off, four groups of 14 of this repository's sources. A probe preloaded into the analyzer sampledmallinfo2()every 250 ms andmalloc_info()every second.--jobs=32, which creates 14 loading workers, against--jobs=1:MALLOC_ARENA_MAX=1, RssAnon after loading comes back within 7 MiB of the--jobs=1level.--jobs=1, the compilations under the lock take 21.0 to 32.9 s of 21.5 to 34.0 s of loading. Reading the bitcode, the only part the workers run in parallel, takes 0.5 to 1.1 s in all. With 14 workers, loading took 2.9 to 5.5 s longer.Set aside:
MALLOC_ARENA_MAXormallopt: a process-wide glibc policy, and each run was 0.5 to 8.7 s longer than at--jobs=1with it;malloc_trimafter loading: glibc-specific, and it would return the memory after the fact, while it accumulates during loading;Verification
Release builds of
57aa98aand of57aa98awith the first commit, from one tree, on a second Linux x86_64 machine, in a container of 4 CPUs and 5 GiB without swap. The same four groups of sources, one run per cell, without and with #239's check, the runs of the two builds interleaved.Memory
#239's check off; MiB; "before / after":
--jobs--jobs=1, both builds take the same path: −12 to +8 MiB, the order of the noise of these samples.--jobs=32, without and with feat(const-param): experimental Clang check of const suggestions, version 2 #239's check, the change removes 1644 to 2451 MiB of RssAnon at the end of loading, which comes back within 13 MiB of its--jobs=1level, and 1579 to 2793 MiB of peak RSS. At--jobs=4, it removes 536 to 984 MiB at the end of loading.--jobs=1at--jobs=32(+69 to +161 MiB at the peak) comes after loading, from the cross-TU summaries, which keep their workers.Time
Seconds, one run each. The pairs at
--jobs=1give the noise: −1.7 to +1.1 s per run.--jobs=32: 1.0 to 3.8 s shorter. Before, it took 2.4 to 5.4 s more than at--jobs=1; after, it stays within −0.2 to +2.0 s of its own--jobs=1.--jobs=32: 1.9 to 9.2 s shorter with feat(const-param): experimental Clang check of const suggestions, version 2 #239's check off, 2.7 to 39.0 s with it on, and 2.3 to 4.7 s less system time. At--jobs=4: −0.9 to +0.1 s, within the noise.--jobs=32than at--jobs=1on this machine: +0.9 to +33.1 s of user CPU time, without page faults or memory pressure. After the change, it stays within −0.8 to +0.3 s of its--jobs=1time. The cause is not established; on the first machine, that effect was +0.4 to +1.4 s.These are memory measurements; the times are given for completeness, not as a speedup.
Outputs
--jobs.GlobalReadBeforeWriteissues, inmatch_patterns(pcre2grep.c) andmy_free(pcre2test.c). That order follows the globals' addresses: it already varied from run to run at--jobs=1before the change, and after it, it also varies at--jobsabove 1. fix(analysis): emit global read-before-write issues in instruction order #284 emits these issues in the order of the instructions.Tests
On
57aa98awith the first commit: CI's build passes, with one warning that was already there, in a unit test file this change does not touch. The unit and ownership tests pass, and the suite passes 2742/2742 as CI runs it (#239's suite, its own checks included). It already compares outputs at--jobs1, 2 and 4.Four analyzers at once
With the change, #239's check on, and the automatic
--jobs(12 on that machine), all four finished. The container peaked at 3.97 of 5 GiB, and the analyzers' summed RssAnon at 3.20 GiB. The build without the change was not run in that configuration.Interaction with #272
#272 (draft) lets compilations run in parallel. This change rests on compilations being serialized, as they are on
mainat1fadba2. The two conflict:README.md: both edit the same sentence on--jobs.#272 stays separate: before it lands, it will have to keep this memory gain and control the concurrency of its compilations. Nothing here measures #272's branch.
CI and code scanning
280ba46, this branch updated withmainatb7afcfb, which holds fix(analysis): emit global read-before-write issues in instruction order #284 (merge refb3bfc2a): every check passes; the release job is skipped, as outside tags.ubuntu-24.04,macos-15andmacos-latest, asmainatb7afcfbdoes. fix(analysis): emit global read-before-write issues in instruction order #284's order check passes there at--jobs=1and at--jobs=4on the three runners, with the inputs now loaded one at a time.b3bfc2a: coretrace-stack-analyzer reported 72 results, one fewer than onmainatb7afcfb, and coretrace-concurrency-analyzer none. New alerts: none.ConstParameterNotModified.Pointernote atsrc/app/AnalyzerApp.cpp:246, in the lambda ofrunParallelWork.mainreports it once per instantiation ofrunParallelWork, five times; this change removes the instantiation for loading. The other results are those ofmain, six of them five lines higher inAnalyzerApp.cpp.main, 65 are open onrefs/pull/285/merge. The one that is not, alert 156, is one of the five at that line.e60a10e(merge ref2d666b2, intomainat9393ef3): every check passed; the suite passed 2915/2915 on the three runners, asmaindid then; code scanning showed the same difference;4ae52d7(merge refc6d7220, intomainat1fadba2): every check passed, and the suite passed 2760/2760 on the three runners, asmaindid then.Reservations
main. The loading code and the compile lock are the same onmainat1fadba2, but memory and time were not measured there. This pull request's CI checks the integration.--compile-ir-cache-dir, which the suite uses in CI) or an input already in LLVM IR (.ll,.bc) does not take the compile lock. Before, such inputs were read in parallel; now they are read one after the other. Their time and memory are not measured.