fix(ci): make the weekly performance gate tolerate runner and network noise by docdyhr · Pull Request #337 · docdyhr/versiontracker

docdyhr · 2026-10-19T09:17:10Z

Summary

The weekly scheduled performance gate (performance.yml → scripts/compare_performance.py) fails on runner and network noise, not code. It failed 3 of the last 6 weeks (09-13, 09-27, 10-18) with no runtime change; no versiontracker/ commits landed between the passing 10-11 baseline and the 10-18 failure.

Two causes, measured on the uploaded result artifacts of run 38182609131 (baseline, 2026-10-11) and run 39174628747 (2026-10-18, attempt 2):

Benchmark	Δ mean (old gate)	Δ best-of-3
`apps` (local)	+30% ❌	+8%
`recommend` (Homebrew API)	+24% ❌	+26%
`outdated` (Homebrew API)	+90% ❌	+61%

The gate used the mean of 3 iterations. One slow iteration inflates it. With best-of-3, apps is within threshold.
recommend and outdated are dominated by live Homebrew API calls. They're slower even at their best on an unchanged codebase, so that gate measures the network.

The full benchmark only runs on schedule and workflow_dispatch. On PRs the required "Performance Testing" job is skipped, so PRs were never blocked; only the weekly run failed.

Changes

scripts/compare_performance.py:
- Compare min_time (best-of-N) when both files have it, falling back to avg_time for older result files.
- recommend and outdated (NETWORK_BOUND) time regressions are reported as WARNING … network-bound, not gated.
- Their memory, and apps time, stay gated.
tests/test_compare_performance.py: 6 tests (there were none), covering:
- best-of-N tolerates one slow iteration;
- a real local best-of-N regression still fails;
- a network-bound time regression only warns;
- a network-bound memory regression still fails;
- the avg_time fallback;
- a replay of the 2026-10-18 numbers, which now passes with two warnings.
CHANGELOG.md: Unreleased → Fixed.

Verification

All 6 new tests pass. All 6 fail on the old script (red-checked by stashing the change).
Full suite: 2839 passed, 16 skipped (2833 before, plus the 6 new).
ruff check, ruff format --check, mypy and the pre-commit hooks are clean.
After merge, run gh workflow run performance.yml once to confirm the next real run on a macOS runner.

🤖 Generated with Claude Code

Summary by Sourcery

Prevent false weekly performance failures by using noise-resistant timing comparisons and excluding network-bound time regressions from the blocking gate.

Bug Fixes:

Make the weekly performance gate resilient to runner noise and live network variability while preserving checks for meaningful local and memory regressions.

Enhancements:

Compare benchmark runtimes using best-of-N results when available, with compatibility fallback to average runtimes.
Report time regressions for network-bound Homebrew benchmarks as non-blocking warnings while continuing to gate their memory usage and local benchmark timing.

Tests:

Add coverage for best-of-N comparisons, legacy result fallback, network-bound warnings, gated memory regressions, and replay of the previously failing scheduled run.

… noise compare_performance.py gated the 3-iteration mean, which one slow iteration inflates, and treated recommend/outdated (dominated by live Homebrew API calls) like local work. The scheduled run failed 3 of the last 6 weeks with no runtime change; on 2026-10-18 (run 39174628747) apps +30%, recommend +24%, outdated +90% against the 2026-10-11 baseline, while best-of-3 was apps +8%, recommend +26%, outdated +61% — the network-bound commands are slower even at their best, i.e. the network, not the code. - Gate on best-of-N (min_time) when both files have it; fall back to avg_time for older result files. - recommend/outdated time regressions are reported as warnings ("network-bound, not gated"); their memory and apps time stay gated. - tests/test_compare_performance.py: 6 tests, including a replay of the 2026-10-18 numbers (passes with two warnings); all 6 fail on the old script. Full suite: 2839 passed, 16 skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

sourcery-ai · 2026-10-19T09:17:15Z

Sorry @docdyhr, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 4 days and 16 hours by commenting @sourcery-ai review. Upgrade to get a review now.

chatgpt-codex-connector · 2026-10-19T09:17:18Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T09:18:38.893187Z	`643291c`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

sourcery-ai · 2026-10-19T09:17:23Z

Reviewer's Guide

The PR makes the scheduled performance gate resilient to runner and Homebrew network noise by comparing best-of-N timings when available, downgrading known network-bound timing regressions to warnings, and retaining gates for local timing and memory regressions. New tests validate the behavior—including a replay of the historical failure—and the changelog documents the fix.

Flow diagram for the resilient performance regression gate

flowchart TD
    A[Load current and baseline results] --> B{Both results have min_time?}
    B -->|Yes| C[Compare min_time]
    B -->|No| D[Compare avg_time]
    C --> E[Check time and memory deltas]
    D --> E
    E --> F{Regression exceeds threshold?}
    F -->|No| G[Pass]
    F -->|Yes| H{Network-bound time regression?}
    H -->|Yes| I[Report WARNING; do not fail]
    H -->|No| J[Report REGRESSION; fail gate]

File-Level Changes

Change	Details	Files
Switch performance comparisons to a noise-resistant best-of-N time metric while preserving compatibility with legacy result files.	Prefer min_time when both baseline and current results provide it. Fall back to avg_time when either result lacks min_time. Continue applying absolute limits and memory regression checks.	`scripts/compare_performance.py`
Exclude known network-bound timing variance from the failure gate without hiding actionable memory regressions.	Report recommend and outdated time regressions as non-failing warnings. Keep apps timing and all benchmark memory regressions gated. Include metric names and network-bound context in diagnostics.	`scripts/compare_performance.py`
Add focused coverage for best-of-N behavior, compatibility, gating boundaries, and the historical failure scenario.	Test tolerance of a single slow iteration and detection of genuine local regressions. Test network timing warnings, memory failures, and avg_time fallback. Replay the 2026-10-18 data and verify a passing exit code with two warnings.	`tests/test_compare_performance.py`
Document the revised weekly performance-gate behavior and rationale.	Record best-of-N comparison and legacy fallback behavior. Document that network-bound timing warnings do not fail the gate while memory remains gated.	`CHANGELOG.md`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

github-actions · 2026-10-19T09:18:02Z

🔒 Security Analysis Report

Security Analysis Report

Generated: Mon Oct 19 09:18:00 UTC 2026
Repository: docdyhr/versiontracker
Commit: c7b3b28

Bandit Security Scan

�[?25l
�[2KWorking... �[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m �[35m  0%�[0m �[36m-:--:--�[0m
�[2KWorking... �[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m �[35m  0%�[0m �[36m-:--:--�[0m
�[2KWorking... �[91m━━━�[0m�[90m╺�[0m�[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m �[35m  8%�[0m �[36m-:--:--�[0m
�[2KWorking... �[91m━━━━━━━━━━━━�[0m�[91m╸�[0m�[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m �[35m 31%�[0m �[36m0:00:01�[0m
�[2KWorking... �[91m━━━━━━━━━━━━━━━━�[0m�[90m╺�[0m�[90m━━━━━━━━━━━━━━━━━━━━━━━�[0m �[35m 41%�[0m �[36m0:00:01�[0m
�[2KWorking... �[91m━━━━━━━━━━━━━━━━━━━━�[0m�[91m╸�[0m�[90m━━━━━━━━━━━━━━━━━━━�[0m �[35m 52%�[0m �[36m0:00:01�[0m
�[2KWorking... �[91m━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m�[91m╸�[0m�[90m━━━━━━━━━━━━━�[0m �[35m 67%�[0m �[36m0:00:01�[0m
�[2KWorking... �[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m�[90m╺�[0m�[90m━━━━━━━━�[0m �[35m 78%�[0m �[36m0:00:01�[0m
�[2KWorking... �[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m�[90m╺�[0m�[90m━━━━━�[0m �[35m 86%�[0m �[36m0:00:01�[0m
�[2KWorking... �[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m �[35m100%�[0m �[33m0:00:00�[0m
�[2KWorking... �[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m �[35m100%�[0m �[33m0:00:00�[0m
�[?25hRun started:2026-10-19 09:18:01.325200+00:00

Test results:
>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector through string-based query construction.
   Severity: Medium   Confidence: Low
   CWE: CWE-89 (https://cwe.mitre.org/data/definitions/89.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b608_hardcoded_sql_expressions.html
   Location: versiontracker/advanced_cache.py:693:24
692	                # Use f-string for better readability
693	                msg = f"Failed to delete from cache {key}: {e}"
694	                raise CacheError(msg) from e

--------------------------------------------------
>> Issue: [B105:hardcoded_password_string] Possible hardcoded password: 'homebrew_token'
   Severity: Low   Confidence: Medium
   CWE: CWE-259 (https://cwe.mitre.org/data/definitions/259.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b105_hardcoded_password_string.html
   Location: versiontracker/audit/models.py:161:21
160	    BUNDLE_ID = "bundle_id"
161	    HOMEBREW_TOKEN = "homebrew_token"
162	    DISPLAY_NAME = "display_name"

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/config.py:174:16
173	                cmd = f"{path} --version"
174	                import subprocess
175	

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b603_subprocess_without_shell_equals_true.html
   Location: versiontracker/config.py:177:29
176	                try:
177	                    result = subprocess.run(cmd.split(), capture_output=True, timeout=2, check=False)
178	                    returncode = result.returncode

--------------------------------------------------
>> Issue: [B110:try_except_pass] Try, Except, Pass detected.
   Severity: Low   Confidence: High
   CWE: CWE-703 (https://cwe.mitre.org/data/definitions/703.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b110_try_except_pass.html
   Location: versiontracker/experimental/analytics.py:645:16
644	                    self.peak_cpu = max(self.peak_cpu, cpu_percent)
645	                except Exception:
646	                    pass
647	                time.sleep(0.05)

--------------------------------------------------
>> Issue: [B101:assert_used] Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
   Severity: Low   Confidence: High
   CWE: CWE-703 (https://cwe.mitre.org/data/definitions/703.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b101_assert_used.html
   Location: versiontracker/handlers/outdated_handlers.py:518:8
517	        # Type assertion: apps and brews cannot be None here due to exit_code checks above
518	        assert apps is not None
519	        assert brews is not None

--------------------------------------------------
>> Issue: [B101:assert_used] Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
   Severity: Low   Confidence: High
   CWE: CWE-703 (https://cwe.mitre.org/data/definitions/703.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b101_assert_used.html
   Location: versiontracker/handlers/outdated_handlers.py:519:8
518	        assert apps is not None
519	        assert brews is not None
520	        apps = _filter_applications(apps, brews, include_brews)

--------------------------------------------------
>> Issue: [B101:assert_used] Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
   Severity: Low   Confidence: High
   CWE: CWE-703 (https://cwe.mitre.org/data/definitions/703.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b101_assert_used.html
   Location: versiontracker/handlers/outdated_handlers.py:539:8
538	        # Type assertion: outdated_info cannot be None here due to exit_code check
539	        assert outdated_info is not None
540	        # Type cast: outdated_info cannot be None here due to exit_code check above

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/homebrew.py:12:0
11	import re
12	import subprocess
13	import time

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/macos_integration.py:12:0
11	import plistlib
12	import subprocess
13	from pathlib import Path

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/menubar_app.py:8:0
7	import logging
8	import subprocess
9	import sys

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/plugins/example_plugins.py:322:16
321	            try:
322	                import subprocess
323	

--------------------------------------------------
>> Issue: [B607:start_process_with_partial_path] Starting a process with a partial executable path
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b607_start_process_with_partial_path.html
   Location: versiontracker/plugins/example_plugins.py:324:25
323	
324	                result = subprocess.run(["brew", "--version"], capture_output=True, text=True, timeout=5)
325	                if result.returncode == 0:

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b603_subprocess_without_shell_equals_true.html
   Location: versiontracker/plugins/example_plugins.py:324:25
323	
324	                result = subprocess.run(["brew", "--version"], capture_output=True, text=True, timeout=5)
325	                if result.returncode == 0:

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/utils.py:16:0
15	import shutil
16	import subprocess
17	import sys

--------------------------------------------------
>> Issue: [B607:start_process_with_partial_path] Starting a process with a partial executable path
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b607_start_process_with_partial_path.html
   Location: versiontracker/utils.py:804:17
803	    try:
804	        result = subprocess.run(["which", "brew"], capture_output=True, text=True, timeout=5)
805	        return result.returncode == 0

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b603_subprocess_without_shell_equals_true.html
   Location: versiontracker/utils.py:804:17
803	    try:
804	        result = subprocess.run(["which", "brew"], capture_output=True, text=True, timeout=5)
805	        return result.returncode == 0

--------------------------------------------------
>> Issue: [B607:start_process_with_partial_path] Starting a process with a partial executable path
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b607_start_process_with_partial_path.html
   Location: versiontracker/utils.py:820:17
819	    try:
820	        result = subprocess.run(["brew", "--prefix"], capture_output=True, text=True, timeout=5)
821	        if result.returncode == 0:

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b603_subprocess_without_shell_equals_true.html
   Location: versiontracker/utils.py:820:17
819	    try:
820	        result = subprocess.run(["brew", "--prefix"], capture_output=True, text=True, timeout=5)
821	        if result.returncode == 0:

--------------------------------------------------
>> Issue: [B603:subprocess_without_shell_equals_true] subprocess call - check for execution of untrusted input.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/plugins/b603_subprocess_without_shell_equals_true.html
   Location: versiontracker/utils.py:853:15
852	    try:
853	        return subprocess.run(command, capture_output=True, text=True, timeout=timeout, check=check)
854	    except subprocess.TimeoutExpired as e:

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/version/batch.py:11:0
10	import logging
11	import subprocess
12	from concurrent.futures import ThreadPoolExecutor

--------------------------------------------------
>> Issue: [B404:blacklist] Consider possible security implications associated with the subprocess module.
   Severity: Low   Confidence: High
   CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
   More Info: https://bandit.readthedocs.io/en/1.9.4/blacklists/blacklist_imports.html#b404-import-subprocess
   Location: versiontracker/version/homebrew.py:18:0
17	import logging
18	import subprocess
19	

--------------------------------------------------

Code scanned:
	Total lines of code: 17221
	Total lines skipped (#nosec): 0
	Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 30

Run metrics:
	Total issues (by severity):
		Undefined: 0
		Low: 21
		Medium: 1
		High: 0
	Total issues (by confidence):
		Undefined: 0
		Low: 1
		Medium: 1
		High: 20
Files skipped (0):

Pip-Audit Results

�[?25l�[32m-�[0m Collecting inputs
�[2K�[32m-�[0m Collecting inputs
�[2K�[32m-�[0m Collecting inputs
�[2K�[32m-�[0m Collecting inputs
�[2K�[32m\�[0m Collecting inputs
�[2K�[32m\�[0m Collecting aiohappyeyeballs (2.7.1)
�[2K�[32m\�[0m Collecting aiohappyeyeballs (2.7.1)
�[2K�[32m\�[0m Auditing aiohappyeyeballs (2.7.1)
�[2K�[32m\�[0m Collecting aiohttp (3.14.5)
�[2K�[32m\�[0m Auditing aiohttp (3.14.5)
�[2K�[32m\�[0m Collecting aiosignal (1.4.0)
�[2K�[32m\�[0m Auditing aiosignal (1.4.0)
�[2K�[32m\�[0m Collecting ast_serialize (0.13.1)
�[2K�[32m\�[0m Auditing ast_serialize (0.13.1)
�[2K�[32m\�[0m Collecting attrs (26.1.0)
�[2K�[32m\�[0m Auditing attrs (26.1.0)
�[2K�[32m\�[0m Collecting bandit (1.9.4)
�[2K�[32m\�[0m Auditing bandit (1.9.4)
�[2K�[32m\�[0m Collecting boolean.py (5.0)
�[2K�[32m\�[0m Auditing boolean.py (5.0)
�[2K�[32m\�[0m Collecting build (1.6.1)
�[2K�[32m\�[0m Auditing build (1.6.1)
�[2K�[32m\�[0m Collecting CacheControl (0.14.4)
�[2K�[32m\�[0m Auditing CacheControl (0.14.4)
�[2K�[32m\�[0m Collecting certifi (2026.10.18)
�[2K�[32m\�[0m Auditing certifi (2026.10.18)
�[2K�[32m\�[0m Collecting cffi (2.1.1)
�[2K�[32m\�[0m Auditing cffi (2.1.1)
�[2K�[32m\�[0m Collecting charset-normalizer (3.5.2)
�[2K�[32m\�[0m Auditing charset-normalizer (3.5.2)
�[2K�[32m\�[0m Collecting coverage (7.16.2)
�[2K�[32m\�[0m Auditing coverage (7.16.2)
�[2K�[32m\�[0m Collecting cryptography (50.0.2)
�[2K�[32m\�[0m Auditing cryptography (50.0.2)
�[2K�[32m\�[0m Collecting cyclonedx-python-lib (11.12.0)
�[2K�[32m\�[0m Auditing cyclonedx-python-lib (11.12.0)
�[2K�[32m\�[0m Collecting defusedxml (0.7.1)
�[2K�[32m\�[0m Auditing defusedxml (0.7.1)
�[2K�[32m\�[0m Collecting docutils (0.23)
�[2K�[32m\�[0m Auditing docutils (0.23)
�[2K�[32m\�[0m Collecting filelock (4.2.4)
�[2K�[32m\�[0m Auditing filelock (4.2.4)
�[2K�[32m\�[0m Collecting frozenlist (1.8.0)
�[2K�[32m\�[0m Auditing frozenlist (1.8.0)
�[2K�[32m\�[0m Collecting id (1.6.1)
�[2K�[32m\�[0m Auditing id (1.6.1)
�[2K�[32m\�[0m Collecting idna (3.20)
�[2K�[32m\�[0m Auditing idna (3.20)
�[2K�[32m\�[0m Collecting iniconfig (2.3.1)
�[2K�[32m\�[0m Auditing iniconfig (2.3.1)
�[2K�[32m\�[0m Collecting jaraco.classes (3.4.0)
�[2K�[32m\�[0m Auditing jaraco.classes (3.4.0)
�[2K�[32m\�[0m Collecting jaraco.context (6.1.2)
�[2K�[32m\�[0m Auditing jaraco.context (6.1.2)
�[2K�[32m\�[0m Collecting jaraco.functools (4.6.0)
�[2K�[32m\�[0m Auditing jaraco.functools (4.6.0)
�[2K�[32m\�[0m Collecting jeepney (0.9.0)
�[2K�[32m\�[0m Auditing jeepney (0.9.0)
�[2K�[32m\�[0m Collecting keyring (25.7.0)
�[2K�[32m\�[0m Auditing keyring (25.7.0)
�[2K�[32m\�[0m Collecting librt (0.16.0)
�[2K�[32m\�[0m Auditing librt (0.16.0)
�[2K�[32m\�[0m Collecting license-expression (30.4.4)
�[2K�[32m\�[0m Auditing license-expression (30.4.4)
�[2K�[32m\�[0m Collecting macversiontracker (1.2.0)
�[2K�[32m\�[0m Auditing macversiontracker (1.2.0)
�[2K�[32m\�[0m Collecting markdown-it-py (4.2.0)
�[2K�[32m\�[0m Collecting markdown-it-py (4.2.0)
�[2K�[32m\�[0m Auditing markdown-it-py (4.2.0)
�[2K�[32m\�[0m Collecting mdurl (0.1.2)
�[2K�[32m\�[0m Auditing mdurl (0.1.2)
�[2K�[32m\�[0m Collecting more-itertools (11.2.1)
�[2K�[32m\�[0m Auditing more-itertools (11.2.1)
�[2K�[32m\�[0m Collecting msgpack (1.2.3)
�[2K�[32m\�[0m Auditing msgpack (1.2.3)
�[2K�[32m\�[0m Collecting multidict (7.1.0)
�[2K�[32m\�[0m Auditing multidict (7.1.0)
�[2K�[32m\�[0m Collecting mypy (2.4.0)
�[2K�[32m\�[0m Auditing mypy (2.4.0)
�[2K�[32m\�[0m Collecting mypy_extensions (1.1.0)
�[2K�[32m\�[0m Auditing mypy_extensions (1.1.0)
�[2K�[32m\�[0m Collecting nh3 (0.3.7)
�[2K�[32m\�[0m Auditing nh3 (0.3.7)
�[2K�[32m\�[0m Collecting packageurl-python (0.17.6)
�[2K�[32m\�[0m Auditing packageurl-python (0.17.6)
�[2K�[32m\�[0m Collecting packaging (26.3)
�[2K�[32m\�[0m Auditing packaging (26.3)
�[2K�[32m\�[0m Collecting pathspec (1.1.1)
�[2K�[32m\�[0m Auditing pathspec (1.1.1)
�[2K�[32m\�[0m Collecting pip (26.2.1)
�[2K�[32m\�[0m Auditing pip (26.2.1)
�[2K�[32m\�[0m Collecting pip_api (0.0.35)
�[2K�[32m\�[0m Auditing pip_api (0.0.35)
�[2K�[32m\�[0m Collecting pip_audit (2.10.1)
�[2K�[32m\�[0m Auditing pip_audit (2.10.1)
�[2K�[32m\�[0m Collecting pip-requirements-parser (32.0.1)
�[2K�[32m\�[0m Auditing pip-requirements-parser (32.0.1)
�[2K�[32m\�[0m Collecting platformdirs (4.13.5)
�[2K�[32m\�[0m Auditing platformdirs (4.13.5)
�[2K�[32m\�[0m Collecting pluggy (1.7.0)
�[2K�[32m\�[0m Auditing pluggy (1.7.0)
�[2K�[32m\�[0m Collecting propcache (0.5.4)
�[2K�[32m\�[0m Auditing propcache (0.5.4)
�[2K�[32m\�[0m Collecting psutil (7.2.2)
�[2K�[32m\�[0m Auditing psutil (7.2.2)
�[2K�[32m\�[0m Collecting py-serializable (2.1.0)
�[2K�[32m\�[0m Auditing py-serializable (2.1.0)
�[2K�[32m\�[0m Collecting pycparser (3.11)
�[2K�[32m\�[0m Auditing pycparser (3.11)
�[2K�[32m\�[0m Collecting Pygments (2.21.0)
�[2K�[32m\�[0m Auditing Pygments (2.21.0)
�[2K�[32m\�[0m Collecting pyparsing (3.3.3)
�[2K�[32m\�[0m Auditing pyparsing (3.3.3)
�[2K�[32m\�[0m Collecting pyproject_hooks (1.3.3)
�[2K�[32m\�[0m Auditing pyproject_hooks (1.3.3)
�[2K�[32m\�[0m Collecting pytest (9.1.1)
�[2K�[32m\�[0m Auditing pytest (9.1.1)
�[2K�[32m\�[0m Collecting pytest-asyncio (1.4.0)
�[2K�[32m\�[0m Auditing pytest-asyncio (1.4.0)
�[2K�[32m\�[0m Collecting pytest-cov (7.1.0)
�[2K�[32m\�[0m Auditing pytest-cov (7.1.0)
�[2K�[32m\�[0m Collecting pytest-mock (3.17.0)
�[2K�[32m\�[0m Auditing pytest-mock (3.17.0)
�[2K�[32m\�[0m Collecting pytest-timeout (2.4.0)
�[2K�[32m\�[0m Auditing pytest-timeout (2.4.0)
�[2K�[32m\�[0m Collecting PyYAML (6.0.3)
�[2K�[32m\�[0m Auditing PyYAML (6.0.3)
�[2K�[32m\�[0m Collecting readme_renderer (46.0)
�[2K�[32m\�[0m Auditing readme_renderer (46.0)
�[2K�[32m\�[0m Collecting requests (2.34.2)
�[2K�[32m\�[0m Auditing requests (2.34.2)
�[2K�[32m\�[0m Collecting requests-toolbelt (1.0.0)
�[2K�[32m\�[0m Auditing requests-toolbelt (1.0.0)
�[2K�[32m\�[0m Collecting rfc3986 (2.0.0)
�[2K�[32m\�[0m Auditing rfc3986 (2.0.0)
�[2K�[32m\�[0m Collecting rich (15.0.0)
�[2K�[32m\�[0m Auditing rich (15.0.0)
�[2K�[32m\�[0m Collecting ruff (0.17.0)
�[2K�[32m\�[0m Collecting ruff (0.17.0)
�[2K�[32m\�[0m Auditing ruff (0.17.0)
�[2K�[32m\�[0m Collecting SecretStorage (3.5.0)
�[2K�[32m\�[0m Auditing SecretStorage (3.5.0)
�[2K�[32m\�[0m Collecting setuptools (84.0.0)
�[2K�[32m\�[0m Auditing setuptools (84.0.0)
�[2K�[32m\�[0m Collecting sortedcontainers (2.4.0)
�[2K�[32m\�[0m Auditing sortedcontainers (2.4.0)
�[2K�[32m\�[0m Collecting stevedore (5.9.1)
�[2K�[32m\�[0m Auditing stevedore (5.9.1)
�[2K�[32m\�[0m Collecting tabulate (0.10.0)
�[2K�[32m\�[0m Auditing tabulate (0.10.0)
�[2K�[32m\�[0m Collecting termcolor (3.3.0)
�[2K�[32m\�[0m Auditing termcolor (3.3.0)
�[2K�[32m\�[0m Collecting tomli (2.5.0)
�[2K�[32m\�[0m Auditing tomli (2.5.0)
�[2K�[32m\�[0m Collecting tomli_w (1.2.0)
�[2K�[32m\�[0m Auditing tomli_w (1.2.0)
�[2K�[32m\�[0m Collecting tqdm (4.70.1)
�[2K�[32m\�[0m Auditing tqdm (4.70.1)
�[2K�[32m\�[0m Collecting twine (7.0.0)
�[2K�[32m\�[0m Auditing twine (7.0.0)
�[2K�[32m\�[0m Collecting types-PyYAML (6.0.12.20260906)
�[2K�[32m\�[0m Auditing types-PyYAML (6.0.12.20260906)
�[2K�[32m\�[0m Collecting typing_extensions (4.16.0)
�[2K�[32m\�[0m Auditing typing_extensions (4.16.0)
�[2K�[32m\�[0m Collecting urllib3 (2.8.0)
�[2K�[32m\�[0m Auditing urllib3 (2.8.0)
�[2K�[32m\�[0m Collecting wheel (0.48.0)
�[2K�[32m\�[0m Auditing wheel (0.48.0)
�[2K�[32m\�[0m Collecting yarl (1.25.1)
�[2K�[32m\�[0m Auditing yarl (1.25.1)
�[2K�[32m\�[0m Auditing yarl (1.25.1)
�[?25h
�[1A�[2K```

codecov · 2026-10-19T09:19:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

#337 switched the gate to best-of-N and warned only on the network-bound benchmarks, but the first post-merge run (39243572918) still failed: `apps` (system_profiler) measured +104% best-of-3 against the 2026-10-11 baseline, after +8% on 2026-10-18, with no runtime change. Wall-clock on shared macOS runners cannot tell a regression from runner noise. - Relative time changes are warnings ("time is advisory"). - The gate fails on what noise cannot produce: a memory regression beyond 20%, or a benchmark whose best iteration (min_time, else avg_time) exceeds the 30 s ceiling — also enforced when there is no baseline. - tests/test_compare_performance.py: 8 tests incl. replays of the 10-18 and 10-19 runs; 6 of them fail on the #337 version. - tests/test_project_consistency.py: locate the regression-check step by its `python scripts/compare_performance.py` invocation, not its display name (the step label now states the new rule). Full suite: 2841 passed, 16 skipped. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

docdyhr merged commit 7288a9e into main Oct 19, 2026
25 checks passed

docdyhr deleted the fix/perf-gate-noise branch October 19, 2026 11:16

docdyhr mentioned this pull request Oct 19, 2026

fix(ci): make wall-clock time advisory in the performance gate #338

Merged

Navigation Menu

Repository navigation

fix(ci): make the weekly performance gate tolerate runner and network noise - #337

fix(ci): make the weekly performance gate tolerate runner and network noise#337
docdyhr merged 1 commit into
mainfrom
fix/perf-gate-noise

docdyhr commented Oct 19, 2026 •

edited by sourcery-ai Bot

Loading

Uh oh!

sourcery-ai Bot commented Oct 19, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

sourcery-ai Bot commented Oct 19, 2026

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

codecov Bot commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

docdyhr commented Oct 19, 2026 • edited by sourcery-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Verification

Summary by Sourcery

Uh oh!

sourcery-ai Bot commented Oct 19, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

sourcery-ai Bot commented Oct 19, 2026

Reviewer's Guide

Flow diagram for the resilient performance regression gate

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

github-actions Bot commented Oct 19, 2026

🔒 Security Analysis Report

Security Analysis Report

Bandit Security Scan

Pip-Audit Results

Uh oh!

codecov Bot commented Oct 19, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

docdyhr commented Oct 19, 2026 •

edited by sourcery-ai Bot

Loading

chatgpt-codex-connector Bot commented Oct 19, 2026 •

edited

Loading