test(ci): cover the commit-lint policy patterns - #597
Merged
Merged
Conversation
Three defects have shipped in commit-lint.yml. #575 could not start at all and took 53 runs to notice. The empty-pattern trap found during #586 would have flagged every commit and passed every branch. #587 failed open on a large commit message. Two of the three were silent, and nothing automated caught any of them. Add a suite that extracts every pattern from the workflow rather than restating it, so the test cannot drift into a second source of truth, and assert the constructs as well as the patterns: no grep -q on the trailer pipeline, an in-step fallback for each pattern, and no workflow_call input default that a pull_request run would ignore. Verified by mutation. Six deliberate regressions were introduced one at a time and every one failed the suite: reintroducing grep -q, removing a fallback, loosening the branch pattern to accept feature-0, dropping the copilot and codex prefixes, widening the trailer pattern to ban human co-authors, and raising the subject cap past 72. Closes #592
This was referenced Sep 2, 2026
ss-o
added a commit
that referenced
this pull request
Sep 2, 2026
Section 4 required a default on every workflow_call input. Two shipped workflows contradict that for two independently correct reasons. zsh-lint.yml declares both its inputs required, and a default on a required input is unreachable because the caller always supplies the value. commit-lint.yml declares workflow_call alongside pull_request. GitHub scopes the inputs context to a reusable or manually triggered workflow, so on a pull_request run it is empty and the declared defaults never apply. #586 removed them for that reason; #597 now asserts no such default exists. Split the bullet into three clauses so the rule matches both shapes. This is a mandatory surface, so as written an agent would have edited both workflows back into defects. Instruction impact review is recorded in the pull request. Closes #598
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #592.
commit-lint.ymlcarries three policy gates whose entire logic is inline shelland regular expressions. Three defects have shipped in it, and nothing
automated caught any of them:
startup_failureworkflow_calldefaults do not apply on apull_requestrun, so both patterns would have been empty: every commit flagged, every branch passedgrep -qunderpipefailfails open on a large commit messageTwo of the three were silent.
actionlintpasses all of them.Design
Every pattern is extracted from the workflow, never restated:
DISALLOWED_TRAILER_PATTERN: "${VAR:=...}"fallbackBRANCH_PATTERN: "${VAR:=...}"fallbackCONVENTIONAL_PATTERNgrep -qEguard in the branch jobA test carrying its own copy of a regex drifts from the thing it claims to
check and then proves nothing. A failed extraction is a hard failure, not a
skip: it means the workflow changed shape and the test has to be re-pointed.
Alongside the pattern cases, three checks assert the constructs, because
that is where two of the three defects lived:
git show | grep -qon the trailer pipeline (fix(ci): commit-lint trailer guard fails open on large commit messages #587)workflow_callinputdefault:that apull_requestrun would ignore (fix(ci): run commit lint directly so the policy gate executes #586)Verification
40 checks pass. More usefully, the suite was mutation-tested: six
deliberate regressions introduced one at a time, each reverted after.
grep -qon the trailer pipelineBRANCH_PATTERNin-step fallbackfeature-0copilot/andcodex/prefixesSix of six. That is the thing this file has never had: evidence the test
detects the failure, not just that it passes against a known-good input.
bash -n,shellcheck -s bash,actionlint, andtrunk checkare clean.Notes
The suite is Bash with a declared floor of 4.0, per
.github/instructions/shell.instructions.md, which requires the dialect to bedeclared before applying language rules.
z-shell/zicarries the same three jobs with its patterns inlined rather thanbehind
workflow_callfallbacks. The extractors here are anchored to thisrepository's shape, so porting the suite needs its own change; ADR-0022 step 5
already flags those two copies as a drift risk. Not attempted here.