Repository navigation
Description
Summary
Two more growth-ratio tests time a scan with the wall clock (time.perf_counter) and fail when the machine is busy. The same cause makes TestPatternLinearity flaky (#1047, nightly #72). Under CPU load:
tests/test_issue_dependencies.py::test_reading_a_hostile_text_grows_linearly_with_its_lengthfailed 3 of 20 runs;tests/test_check_audit_event_catalog.py(thetest_scaling_is_linear_on_adversarial_inputtest) failed 1 of 20.
Neither pattern is super-linear: the tests mistake scheduler noise for growth.
Evidence
Measured on development at 735a02f (macOS, 18 cores, Python 3.13). 36 busy-loop processes ran during the measurement, with a load average of 41-81. Each selected test was run 20 times with pytest -k <test> -o addopts="", except the secrets test, which ran 10 times because one run takes about 70 s.
| Test | Limit | Failed | Failing values |
|---|---|---|---|
test_issue_dependencies.py test_reading_a_hostile_text_grows_linearly_with_its_length |
about 30x for a 10x input | 3/20 | inline elements 40.8x, a long reference list 31.4x, a keyword and a long space run 35.9x |
test_check_audit_event_catalog.py test_scaling_is_linear_on_adversarial_input |
30x for a 10x input | 1/20 | [0.000304, 0.001519, 0.009355] s (30.8x) |
test_phase12_review_fixes.py heading pattern |
3x the input ratio | 0/20 | |
test_data_audit_phase12.py unclosed BEGIN markers |
3x the size ratio | 0/10 | |
test_check_source_path_refs.py TestPatternLinearity (#1047) |
20x | 0/20 in this run | 25.4x in the nightly run of 2026-09-22, 21.0x in #72, 42.8x locally in #1047 |
The other timing tests use absolute ceilings (100 ms or 1 s on 10K input), or apply the ratio only when the baseline is at least 1 ms. They stay far from their limits, and nothing here suggests changing them.
Why it matters
tests/test_issue_dependencies.pyruns in the full suite and again in CI's coverage step. That step enforces the 90 % floor fortools/issue_dependencies.py, a workflow script with a token that can write to issues. A flaky failure there blocks unrelated pull requests.- The gauntlet in
CLAUDE.mdruns it twice, so a busy development machine fails the gauntlet for no reason, as TestPatternLinearity still fails on a busy machine: the wall-clock ratio check flakes (42.8x against a 20x limit) #1047 describes.
Proposed fix
Apply the approach decided for #1047 to these two tests:
- Measure CPU time (
time.process_time()) instead of wall time, and keep the median. - Calibrate the batch so that one timed sample lasts at least
max(0.025, 8 * time.get_clock_info("process_time").resolution)seconds. On Windows the CPU clock ticks at about 15.6 ms, andpublish.ymlruns the suite onwindows-latest.
Per test:
test_issue_dependencies.py:_seconds_per_calluses a fixedrepeat(20 for the small text, 2 for the large one). Replace it with a calibrated repeat count per input, and keep the 3x allowance.test_check_audit_event_catalog.py: time a calibrated batch ofsearchcalls per size instead of one call, and keep the 30x limit and the 1 s floor.
Do not add retries, skips or looser limits.
Acceptance criteria
- Both tests measure CPU time with a calibrated batch, as above.
- Each test gains a negative control showing that a quadratic reader or pattern still fails the ratio check, like
test_quadratic_scan_is_still_rejectedintests/test_check_source_path_refs.py. - Evidence in the pull request: 20 runs of each test under CPU load (for example 40 busy loops:
for i in $(seq 1 40); do timeout 300 python3 -c 'while True: pass' & done) pass, with the min/max ratio before and after. -
python3 -m coverage run --source=tools -m pytest -q -p no:cov -o addopts="" tests/test_issue_dependencies.py tests/test_assign_on_comment.py tests/test_triage_gate.pystill reaches the 90 % floor.
Context
Same root cause as #1047 (which keeps its own scope: TestPatternLinearity only) and #72. Do the two together, or do #1047 first, so that all three tests share one approach. docs/standards/regex.md ("median of N runs") stays true.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTodo
Activity
cemililik commentedon Oct 22, 2026
Triage (2026-10-22): accepted. Low · Wave 4 · parent #875 (Wave 4 · Module sweep), Size S. The evidence and the proposed fix are in the description. It is not blocked and not a Phase 16 unit; it uses the same approach as #1047, so it is easiest to do once #1047 has landed.