Repository navigation
ci: divide and simplify the workflows into scripts, shared heads and smaller files #1967
Copy link
Copy link
Closed
Labels
ciPull requests that change CI or workflow configuration (ci: subject prefix)Pull requests that change CI or workflow configuration (ci: subject prefix)
Milestone
Description
This is a review of .github/workflows/ at a1cd9efbc. It proposes a sequence of small PRs, not a rewrite. The reports, with file:line evidence, are unit-tests.md, other-workflows.md and cross-cutting.md in the audit set. #1958's phases come first, since they edit the same files.
Findings
unit-tests.ymlis 2056 lines, and 43% of it is comments, mostly history narrative. Moving that narrative toworkflows.rstshrinks the file more than any restructure.- Splitting files.
required-checkscannotneeds:a job in another file, so separate top-level workflows would mean editing the ruleset. The workable split is reusable sub-workflows called fromunit-tests.yml. That goes last. - Repeated inline shell. These blocks are better as
util/scripts than as composite actions, because composite inner steps cannot taketimeout-minutes, and the apt caps double as hang detectors:- the
changesclassifier (byte-identical in two workflows); - the KTLO/experimental summary (5 copies);
- the annotated install (3 full copies, 4 bare);
- the apt cache block;
- sample generation (3 copies);
create-release.yml's tag check, release upload, evidence checks andrelease_status(about 550 lines of shell).
- the
cron-conda,cron-vendoranddeploy-pagesrepeat the same schedule gate andverdicthead, which fits one reusable workflow. The conda build leg is duplicated betweencreate-releaseandcron-conda.- The Saturday
gateruns three times, often on one commit: Pages at 02:00, Vendor and Conda at 10:00. A single weekly gate that the three listen on would save two gate runs a week. - Workflow pins. The workflow-pinning tests re-parse YAML on every call with the pure-Python loader. One module spends 33.6 of its 34.2 s parsing. A cached loader in a shared
tests/_workflows.pywould cut about 200 s of test time, and would replace about nine hand-written scanners. - File-size caps. Only
project-tests(ulimit -f) and the experimental legs (prlimit) cap file size.test,integration, parity, engines,gateand the ordering legs have no cap. - Smaller items:
- unreachable Windows conda steps;
- CodeQL template boilerplate;
- no top-level
permissionsincreate-release.ymland CodeQL; - about 37
file.yml:Nline citations in the docs that every change shifts; - nothing runs actionlint (1.7.12 handles the anchors; 1.7.7 does not).
Proposed order
- Cached workflow loader for the pins (test-only).
util/scripts: KTLO summary,changesclassifier, coverage report, apt, install, samples.- File-size caps on every pytest step.
- actionlint in Lint.
- Comment trim, job by job.
- Reusable cron head; weekly gate.
- Optional sub-workflow split of
unit-tests.yml.
Items 2, 3 and 7 follow #1958 phase 1.
Decisions
- A. Cite workflow steps by job and step name instead of
file.yml:N. - B. One weekly gate instead of three Saturday gate runs (changes the design recorded in
workflows.rst). - C. Move
create-release.yml's shell to scripts, and share the conda leg withcron-conda. The release workflow is yours, so this is your call. - D. Let workflow pins require PyYAML, retiring the text scanners.
Security items are in a separate issue.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
ciPull requests that change CI or workflow configuration (ci: subject prefix)Pull requests that change CI or workflow configuration (ci: subject prefix)
Projects
- StatusShow more project fieldsDone
Activity
JarryShaw commentedon Oct 20, 2026
A. yes
B. might still want each workflow focus on one single task tho. one big task can span multiple workflows but one workflow should only take one task.
C. sure, reuse and modularise if good.
D. sure.
JarryShaw commentedon Oct 20, 2026
Noted:
create-release.yml, and reuse what fits.Under B, the weekly gate becomes its own workflow whose one task is gating. Vendor, Conda and Pages keep their own tasks, and each listens on the gate. Starting with step 1, the cached workflow loader, which doesn't touch #1958's files. The rest follows #1958 phase 1.
JarryShaw commentedon Oct 20, 2026
Step 1 is on
mainasd850b8097, a test-only push.tests/_workflows.pyparses each workflow text once per process, usingCSafeLoaderwhere available, and hands each caller a deep copy. The cache is keyed on the text, so a doctored copy parses afresh. The sharedworkflow()helper and the loaders of nine pins now use it, with no assertion changes. Measured locally,test_ktlo_legs_stay_green_1672drops from 34.7 s to 0.7 s, and the migrated and helper-using modules from about 143 s to 73 s.test: parse each workflow once and share the loader across the workfl…
JarryShaw commentedon Oct 20, 2026
Blocked until #1973 merges. Steps 2–7 move
unit-tests.ymlshell intoutilscripts, and #1973 is still editing those same steps.107 remaining items
ci(release): move the conda legs' shell into util/ scripts (#1967)
ci(release): move the conda legs' shell into util/ scripts (#1967)
JarryShaw commentedon Oct 23, 2026
Audit of the merged migrations against the inline rule (approved): no clear revert. Each script is duplicated, has real logic or a bug history, and none has a hand-run use. Three were borderline:
apt_packages.py seed: back inline. Its original shell was 2 lines (sudo cpplus an echo).release_vendor_update.py: back inline, at 19 lines in a single place. That also drops theutil/checkout the job needed only for the script.release_tags.py owner: kept. It shares the tag-resolve code withcheck, which three jobs use.A PR for the two reverts follows.
ci(release): move the conda legs' shell into util/ scripts (#1967) (#…
ci: put two trivial util/ steps back inline (#1967)
ci: put two trivial util/ steps back inline (#1967) (#2092)
ci(release): move Reset Build Number into util/release_reset_build.py (…
ci(release): move Reset Build Number into util/release_reset_build.py (…
JarryShaw commentedon Oct 23, 2026
Done; closing. #2093 merged at 19:16Z (
d2b9b324b), so every item of the plan has landed or been dropped by ruling.util/scripts: ci: move the repeated workflow shell into util scripts (#1967) #1995, ci: regenerate sample captures through util/regenerate_samples.py (#1967) #2029, ci: install through util/pip_install.py (#1967) #2031, ci: install through util/pip_install.py in experimental and Compat (#1967) #2032, ci: run the apt steps through util/apt_packages.py (#1967) #2034, ci(release): run the tag checks and the release upload from util/ scripts (#1967) #2065, ci(release): run the release status and vendor-update reads from util/ scripts (#1967) #2070, ci(release): run version_check's tier and completeness reads from util/ scripts (#1967) #2081, ci(release): move the conda legs' shell into util/ scripts (#1967) #2089 and ci(release): move Reset Build Number into util/release_reset_build.py (#1967) #2093.util/only when it is duplicated or has real logic.vendor_updatesteps back inline, under that rule.Lintis a required check (ci: make Lint's actionlint step blocking (#1967) #2043).unit-tests.ymlstays one workflow.file.yml:Ncitation is left. Decision C: the conda leg is shared with cron-conda (ci(release): move the conda legs' shell into util/ scripts (#1967) #2089), and the dead Windows conda steps are gone (ci(conda): drop the unreachable Windows conda steps (#1967) #2055).docs(tests): seventh whole-repo pass over the tests (#1103) (#2267)
test: name tests, constants and labels after the behaviour (#2325)