Skip to content

Two more wall-clock growth-ratio tests flake under CPU load: test_issue_dependencies (3 of 20 runs) and the audit-event-catalog scan (1 of 20) #1059

Open

Description

@cemililik

Summary

Two more growth-ratio tests time a scan with the wall clock (time.perf_counter) and fail when the machine is busy. The same cause makes TestPatternLinearity flaky (#1047, nightly #72). Under CPU load:

  • tests/test_issue_dependencies.py::test_reading_a_hostile_text_grows_linearly_with_its_length failed 3 of 20 runs;
  • tests/test_check_audit_event_catalog.py (the test_scaling_is_linear_on_adversarial_input test) failed 1 of 20.

Neither pattern is super-linear: the tests mistake scheduler noise for growth.

Evidence

Measured on development at 735a02f (macOS, 18 cores, Python 3.13). 36 busy-loop processes ran during the measurement, with a load average of 41-81. Each selected test was run 20 times with pytest -k <test> -o addopts="", except the secrets test, which ran 10 times because one run takes about 70 s.

Test Limit Failed Failing values
test_issue_dependencies.py test_reading_a_hostile_text_grows_linearly_with_its_length about 30x for a 10x input 3/20 inline elements 40.8x, a long reference list 31.4x, a keyword and a long space run 35.9x
test_check_audit_event_catalog.py test_scaling_is_linear_on_adversarial_input 30x for a 10x input 1/20 [0.000304, 0.001519, 0.009355] s (30.8x)
test_phase12_review_fixes.py heading pattern 3x the input ratio 0/20
test_data_audit_phase12.py unclosed BEGIN markers 3x the size ratio 0/10
test_check_source_path_refs.py TestPatternLinearity (#1047) 20x 0/20 in this run 25.4x in the nightly run of 2026-09-22, 21.0x in #72, 42.8x locally in #1047

The other timing tests use absolute ceilings (100 ms or 1 s on 10K input), or apply the ratio only when the baseline is at least 1 ms. They stay far from their limits, and nothing here suggests changing them.

Why it matters

Proposed fix

Apply the approach decided for #1047 to these two tests:

  • Measure CPU time (time.process_time()) instead of wall time, and keep the median.
  • Calibrate the batch so that one timed sample lasts at least max(0.025, 8 * time.get_clock_info("process_time").resolution) seconds. On Windows the CPU clock ticks at about 15.6 ms, and publish.yml runs the suite on windows-latest.

Per test:

  • test_issue_dependencies.py: _seconds_per_call uses a fixed repeat (20 for the small text, 2 for the large one). Replace it with a calibrated repeat count per input, and keep the 3x allowance.
  • test_check_audit_event_catalog.py: time a calibrated batch of search calls per size instead of one call, and keep the 30x limit and the 1 s floor.

Do not add retries, skips or looser limits.

Acceptance criteria

  • Both tests measure CPU time with a calibrated batch, as above.
  • Each test gains a negative control showing that a quadratic reader or pattern still fails the ratio check, like test_quadratic_scan_is_still_rejected in tests/test_check_source_path_refs.py.
  • Evidence in the pull request: 20 runs of each test under CPU load (for example 40 busy loops: for i in $(seq 1 40); do timeout 300 python3 -c 'while True: pass' & done) pass, with the min/max ratio before and after.
  • python3 -m coverage run --source=tools -m pytest -q -p no:cov -o addopts="" tests/test_issue_dependencies.py tests/test_assign_on_comment.py tests/test_triage_gate.py still reaches the 90 % floor.

Context

Same root cause as #1047 (which keeps its own scope: TestPatternLinearity only) and #72. Do the two together, or do #1047 first, so that all three tests share one approach. docs/standards/regex.md ("median of N runs") stays true.

Activity

  1. added
    bugSomething isn't working
    severity: lowMinor defect, inaccuracy or rough edge
    wave: 4Roadmap wave 4: low / info / watch, when touching the module
    on Oct 22, 2026
  2. added
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    on Oct 22, 2026
  3. cemililik commented on Oct 22, 2026

    @cemililik
    CollaboratorAuthor

    Triage (2026-10-22): accepted. Low · Wave 4 · parent #875 (Wave 4 · Module sweep), Size S. The evidence and the proposed fix are in the description. It is not blocked and not a Phase 16 unit; it uses the same approach as #1047, so it is easiest to do once #1047 has landed.

  4. removed
    needs-triageNot triaged yet: every new issue starts with it; only a maintainer removes it
    on Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: dev-toolingCode area: dev-toolingbugSomething isn't workingseverity: lowMinor defect, inaccuracy or rough edgewave: 4Roadmap wave 4: low / info / watch, when touching the module

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions