Skip to content

test(ci): cover the commit-lint policy patterns - #597

Merged
ss-o merged 1 commit into
mainfrom
feature-592
Sep 2, 2026
Merged

test(ci): cover the commit-lint policy patterns#597
ss-o merged 1 commit into
mainfrom
feature-592

Conversation

@ss-o

@ss-o ss-o commented Sep 2, 2026

Copy link
Copy Markdown
Member

Closes #592.

commit-lint.yml carries three policy gates whose entire logic is inline shell
and regular expressions. Three defects have shipped in it, and nothing
automated caught any of them:

Issue Defect How it was found
#575 Workflow never started; 53 of 53 runs startup_failure Someone read the run conclusions
#586 workflow_call defaults do not apply on a pull_request run, so both patterns would have been empty: every commit flagged, every branch passed Reading the diff
#587 grep -q under pipefail fails open on a large commit message Reading the diff

Two of the three were silent. actionlint passes all of them.

Design

Every pattern is extracted from the workflow, never restated:

Extracted From
DISALLOWED_TRAILER_PATTERN the : "${VAR:=...}" fallback
BRANCH_PATTERN the : "${VAR:=...}" fallback
CONVENTIONAL_PATTERN its assignment
Allowed automation prefixes the grep -qE guard in the branch job

A test carrying its own copy of a regex drifts from the thing it claims to
check and then proves nothing. A failed extraction is a hard failure, not a
skip: it means the workflow changed shape and the test has to be re-pointed.

Alongside the pattern cases, three checks assert the constructs, because
that is where two of the three defects lived:

Verification

40 checks pass. More usefully, the suite was mutation-tested: six
deliberate regressions introduced one at a time, each reverted after.

Mutation Result
Reintroduce grep -q on the trailer pipeline caught
Remove the BRANCH_PATTERN in-step fallback caught
Loosen the branch pattern to accept feature-0 caught
Drop the copilot/ and codex/ prefixes caught
Widen the trailer pattern so it bans human co-authors caught
Raise the subject-length cap past 72 caught

Six of six. That is the thing this file has never had: evidence the test
detects the failure, not just that it passes against a known-good input.

bash -n, shellcheck -s bash, actionlint, and trunk check are clean.

Notes

The suite is Bash with a declared floor of 4.0, per
.github/instructions/shell.instructions.md, which requires the dialect to be
declared before applying language rules.

z-shell/zi carries the same three jobs with its patterns inlined rather than
behind workflow_call fallbacks. The extractors here are anchored to this
repository's shape, so porting the suite needs its own change; ADR-0022 step 5
already flags those two copies as a drift risk. Not attempted here.

Three defects have shipped in commit-lint.yml. #575 could not start at all and
took 53 runs to notice. The empty-pattern trap found during #586 would have
flagged every commit and passed every branch. #587 failed open on a large
commit message. Two of the three were silent, and nothing automated caught any
of them.

Add a suite that extracts every pattern from the workflow rather than
restating it, so the test cannot drift into a second source of truth, and
assert the constructs as well as the patterns: no grep -q on the trailer
pipeline, an in-step fallback for each pattern, and no workflow_call input
default that a pull_request run would ignore.

Verified by mutation. Six deliberate regressions were introduced one at a time
and every one failed the suite: reintroducing grep -q, removing a fallback,
loosening the branch pattern to accept feature-0, dropping the copilot and
codex prefixes, widening the trailer pattern to ban human co-authors, and
raising the subject cap past 72.

Closes #592
@ss-o
ss-o requested a review from a team as a code owner September 2, 2026 19:14
@ss-o
ss-o merged commit 413307b into main Sep 2, 2026
10 checks passed
@ss-o
ss-o deleted the feature-592 branch September 2, 2026 19:16
ss-o added a commit that referenced this pull request Sep 2, 2026
Section 4 required a default on every workflow_call input. Two shipped
workflows contradict that for two independently correct reasons.

zsh-lint.yml declares both its inputs required, and a default on a required
input is unreachable because the caller always supplies the value.

commit-lint.yml declares workflow_call alongside pull_request. GitHub scopes
the inputs context to a reusable or manually triggered workflow, so on a
pull_request run it is empty and the declared defaults never apply. #586
removed them for that reason; #597 now asserts no such default exists.

Split the bullet into three clauses so the rule matches both shapes. This is a
mandatory surface, so as written an agent would have edited both workflows back
into defects. Instruction impact review is recorded in the pull request.

Closes #598
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(ci): cover the commit-lint policy patterns

1 participant