Repository navigation
ci: periodic CI-duration review, and optimisations as the suite grows #1538
Description
The owner asked for a periodic review of CI time and for optimisations as the test suite grows. This is the first review, measured over the last ~55–150 Unit Tests runs (2026-10-05 to 10-09).
Where the time goes:
- A PR becomes mergeable 14.4 min after its push (median; p90 19.6).
- The critical path is the
Python 3.14coverage leg. It finished last in 43 of 55 PR runs, and itsRun unit testsstep takes 551 s against 332–372 s on 3.10–3.13 (1.57× paired, rising). - 71% of local unit-test time is per-class
pcapkitre-imports. Tests grew 33% since test: re-import pcapkit only where isolation needs it (#1065) #1070 while test time grew ~90%. - The runner pool is capped at 20 concurrent jobs, and Unit Tests uses 92% of it. One PR push queues 28 jobs; one
mainpush queues 38. - 18% of runner time went to superseded PR runs; about half of those were superseded by rebases.
Projections (estimates): the 20-min step cap on Plain unittest ordering (protocols (rest)) is hit in ~6–18 days, and the 3.14 leg's 45-min cap in ~10–30 days.
Starting now:
- P3: JUnit timings and
--durationsfrom CI. - Periodic review: a weekly
ci-timing.ymlwith a stdlibutil/ci_timing.py. It reports job, step and test trends against thresholds anchored to today's numbers, and fails (emailing the owner) when a job's p90 reaches 50% of its cap or a projection reaches the cap within 14 days.
Needs the owner's call, with savings as re-derived by the verifiers:
| # | Proposal | Re-derived saving | Verdict |
|---|---|---|---|
| P1 | Shard the coverage leg in two, plus a combine job | ~1.75 min at the PR median, ~0 on push; 40% of runs slower; reverses #1063 | unsound as written |
| P2 | Stop the per-test re-imports in 7 files (13 classes; two are documented on purpose) | ~22–40 s serial, ~1–3 runner-min per PR | sound with changes |
| P4 | Make the 11 ordering cells needs: test, or move them to schedule-only |
(b) changes what gates Pages, Conda and release; reverses #1052 H | sound with changes |
| P6 | Engines 3.15 cell on main pushes only |
~3 runner-min per PR | unverified |
| P7 | Lazy TCP/UDP app-type registries (33% of import time) | ~35–55 s per leg | unverified |
| P8 | Drop Engines (DPKT), which repeats test_dpkt_unit.py |
~3 runner-min per run | unverified |
P5 (rebase only when needed) is process, not CI, and the agents have already adopted it: 27 of 43 merges in 24 h landed BEHIND through the admin bypass, so a rebase pushed only to clear BEHIND wastes runner time.
Activity
ci: weekly CI timing review and per-test timings from the unit legs (#…
ci: weekly CI timing review and per-test timings from the unit legs (#…
JarryShaw commentedon Oct 10, 2026
All is good and accepted
JarryShaw commentedon Oct 10, 2026
Recorded: the owner accepted all of the remaining proposals. #1542 has already merged P3 and the weekly review. The rest, and how each will land:
needs: [test]and keep running on every push, so they still gate Pages, Conda, Vendor and Release. I chose (a) because (b) would stop ordering from gating publishing, and nothing in the acceptance reverses that. Say so if you want (b).mainpushes only.Engines (DPKT)job is dropped.test_logging. The deliberate purge intest_const_enum_builtin_parity.pystays.unit-tests.yml.Workers are starting now on three PRs: one for P4, P6 and P8 together, one for P2, and one for P7. Each PR re-measures its own saving rather than reusing the estimates here.
ci: queue the ordering cells after the unit legs, run Engines 3.15 on…
test: keep the per-class pcapkit import in 12 classes that re-importe…
JarryShaw commentedon Oct 10, 2026
P1 (coverage shard): new measurements say it would make CI slower. Should it still be built?
The worker re-measured P1 before building it:
Python 3.14finished last in only 7. I re-counted this myself. Splitting it cannot help the other 8.tests/protocols+tests/projectis 50.5% of the leg's test time.Building it would also need four files beyond the brief:
util/ci_timing.py:COVERAGE_JOBand thejunit-unit-3.14artifact are both single-leg.tests/project/test_ci_timing.py:680..github/coverage.toml.docs/source/contributing/conventions/test-coverage.rst.Options:
ci-timing.ymlreport flags if the 3.14 leg becomes the bottleneck again.I recommend 1. P2, P4, P6 and P8 have merged, and P7 is in review as #1560.
perf(const): load the TCP and UDP app-type registries on first use (#…
JarryShaw commentedon Oct 10, 2026
take 1
perf(const): load the TCP and UDP app-type registries on first use (#…
JarryShaw commentedon Oct 10, 2026
Recorded: you chose option 1, so P1 (the coverage shard) is dropped. If the 3.14 leg becomes the bottleneck again, the weekly
ci-timing.ymlreport will flag it.That completes this review:
2770404b9).I'm closing this. The next periodic review will open its own issue when the weekly report trips.