Repository navigation
perf: reduce preprocessing copies and add PGO replay tooling - #220
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native preprocessing copied cached directive tokens and repeatedly cloned macro argument and token-paste inputs. Borrow cached directives until line remapping requires a copy, record argument uses when defining macros, and move expanded tokens on their last use. Regression coverage preserves
#linelocations across shared headers, raw spelling for#/##, and single evaluation of repeated__COUNTER__arguments.Add
tools/benchmarks/pgo.pyto replay recorded C compilations, collect and merge profiles with the selected Rust toolchain, and produce an optional PGO compiler without replacing the normal release binary. Document the workflow and five reproducible benchmark experiments.Validation on Linux x86_64 with Rust 1.86.0 and Clang 15:
cargo fmt --checkcargo clippy --all-targets --all-features -- -D warningscargo test: 52 library tests, 2,105 integration tests, one doctest passed; two integration tests ignored.The copy changes reduced batch dependency-scan allocation requests by 16% and median elapsed time by 6.4%. The additional PGO corpus experiment compared the preserved curl-only compiler (477 training compilations) with curl/Git/Redis training (1,229 compilations):
Curl/Git differences are smaller than run-to-run variation; Redis shows a small improvement. Full conditions, raw samples and compiler hashes are included. Benchmark binaries use the same
7804347-based source plus copy changes and predate newer record-member changes on main; the rebased PR has been retested, but these timings do not measure its newer base. macOS/AArch64 were not measured locally. Profiles and binaries remain ignored local artifacts.The saved mixed-corpus PGO compiler was also remeasured against installed Clang 15.0.7, with three clean builds per compiler/application at
-O2 -g0,make -j4, using Dynamite's default external Clang IR backend. The earlier-O0/LLVM-tools-backend results are a separate configuration.All 18 additional builds and application smoke checks passed. Conditions and raw samples and a JSON snapshot include commands, hashes and validation. This uses the same saved
7804347-based PGO binary as above; it does not measure the rebased head or executable runtime performance. Curl shares one Dynamite-generated configuration between compilers. Only reports changed in this follow-up; snapshot arithmetic, report links andgit diff --checkwere verified.A matching
-O0 -g0comparison keeps the saved compiler binaries, default Clang backend andmake -j4parallelism fixed:All 18 additional builds and smoke checks passed. The -O0 report preserves every sample and compares -O0 with the previous -O2 session. The first Clang curl build (28.53 s) and first PGO Redis build (11.63 s) had substantial wall-time variation; those samples are retained and their cause was not isolated. The prior 10.73 s Redis result used a different Dynamite revision and LLVM tools backend, so it is not a direct PGO comparison. No compiler code changed; compiler hashes, build options, snapshot arithmetic, report links and diff formatting were checked.