Repository navigation
test: cache scanners per test binary to cut CI runtime - #191
hermanngeorge15 merged 3 commits into
Conversation
hermanngeorge15
left a comment
There was a problem hiding this comment.
Reviewed and independently verified. This is the #188 fix and it does what it claims.
Measured on macOS (arm64), same machine, same command, cargo test --locked:
| branch | wall time |
|---|---|
main |
11m 02s |
| this PR | 3m 26s |
3.2x, both green, exit 0. cargo fmt --all -- --check and cargo clippy --all-targets --locked -- -D warnings are both clean on the branch. That corroborates your 29.90 -> 4.32 min on Windows and 4m 38s on CI — three different machines, the same shape of result.
On the design, the parts I checked specifically:
tests/common/mod.rs—OnceLock<Scanner>per integration-test binary is the right granularity. Rust builds one binary pertests/*.rs, so this is one compile per binary rather than one global, which keeps the suites independent.pattern_relaxed_control_test.rscorrectly keeps a separate cached relaxed scanner. Sharing the shipped one would have broken GATE-05.perf_regression_test.rskeeps its deliberate fresh-compilation sampling. That was the one place a naive cache would have silently invalidated the measurement, and you avoided it.tests/pattern_policy_test.rswalkstests/*.rsnon-recursively, sotests/common/mod.rssitting in a subdirectory does not disturb the test-case policy counts.
CI is red on PERF-01 only, and that is not your fault — see my reply on #188. Nothing in this PR touches production code or patterns.
One non-blocking suggestion. The diff removes the measured timing breakdown comment in ci.yml (the 18.1-minute test leg and the per-binary figures). Those numbers are the evidence for why the ceiling is 60 minutes, and they are also the "before" half of the proof that this PR worked. Consider keeping them, relabelled as the pre-#188 baseline, next to your new note.
Approving. I want the gate fixed on main first so this merges green rather than merging over a red badge.
8308a58 to
2af1a95
Compare
2af1a95 to
e53c2e2
Compare
The hook runs fmt, clippy and the full test suite after EVERY turn, regardless of what the turn touched. On a docs-only session that is pure cost: before #191 it was a guaranteed stall, because `cargo test` took longer than the hook's own 900s timeout (11m02s locally, 18.1 min on CI) and was killed without reporting anything. #191 cut that to 3m01s so the gate completes now, but three minutes of recompiling after editing a Markdown file still buys no information. Gate on whether anything fmt, clippy or the tests actually read has changed: *.rs, patterns/, tests/, Cargo.toml, Cargo.lock, build.rs and rust-toolchain.toml, in the working tree first and then in commits not yet on origin/main, so a feature branch stays gated after its work is committed. tests/ is watched because corpus files drive the recall numbers even when no pattern moves. Verified: clean tree exits 0 in 0s; touching src/lib.rs, patterns/core/*.yaml or a tests/corpus file each still runs the full gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The recall, pattern, example, and corpus tests repeatedly compiled all 79 embedded patterns for each payload. They now share a
OnceLock<Scanner>per integration-test binary throughtests/common/mod.rs; relaxed-pattern controls retain a separate cached scanner. The four suites named in #188 use the cache, along with the other repeated scan helpers.The static cache confirms
ScannerisSend + Sync. A concurrent test verifies shared identity and that suppressed, clean, and attack reports remain independent. Synthetic probe scanners and the performance test's deliberate fresh-compilation samples are preserved. Production code, patterns, dependencies, and the completeEXPECTEDrecall table are unchanged.Local before/after timings
Windows, Rust stable, unoptimized test profile, same machine and default test parallelism, one complete run each:
Both commands were
cargo test --locked --no-fail-fast -- --skip findings_name_the_repository_path_not_the_staging_copy, excluding the existing Windows hook path-separator assertion. Command elapsed includes compilation (baseline 61 s, candidate 5.53 s); test-binary timings above exclude it. The baseline completed with only the Windows CRLF catalogue equality failure; the candidate passed after local LF normalization, which creates no committed catalogue diff. The unchanged prefilter-equivalence suite remains a substantial runtime cost (150.50 s before, 139.13 s after). These are measured local results, not a forecast for hosted CI.Validation and CI ceiling
cargo fmt --all -- --checkandcargo clippy --locked --all-targets -- -D warningspass.Closes #188.