Skip to content

Add CI workflow to run Pester suite on self-hosted Windows runner - #222

Open
J-MaFf wants to merge 4 commits into
mainfrom
feat/pester-ci-workflow
Open

Add CI workflow to run Pester suite on self-hosted Windows runner#222
J-MaFf wants to merge 4 commits into
mainfrom
feat/pester-ci-workflow

Conversation

@J-MaFf

@J-MaFf J-MaFf commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Fixes #221

Changes

  • Adds .github/workflows/test.yml: runs the non-integration Pester suite (./tests/run-tests.ps1) on every pull request, using the self-hosted Windows runner already connected to this repo (GH-RUNNER-FLAUI) but previously unused by any workflow.
  • Bootstraps the Pester 5+ module if missing (idempotent — no-op if already present).
  • Fails the job on any test failure — no swallowed exit codes.
  • timeout-minutes: 20 — there's exactly one connected runner, so an unbounded hang would block every subsequent workflow run indefinitely.
  • Does not add a required status check. While blueprinting this, running the full suite directly on a real Windows dev machine reproduced a real failure + hang in switch_to_main's "Should fail when there are uncommitted changes" test (apparent cross-file state pollution across the 10 test files). Gating merges now would risk blocking everything on a suite that isn't provably stable — that diagnosis is tracked separately as bead gitconfig-cm6.
  • CHANGELOG.md / STATUS.md updated.

Spec: specs/archive/add-pester-ci-workflow.md (blueprint → forge, same as #220).

Testing

  • forge generator/evaluator loop: all 9 acceptance criteria (C1-C9, C-final) passed on round 1.
  • Evaluator executed the Pester-bootstrap step's script directly in real pwsh for both the "already installed" and "missing" branches, and empirically confirmed non-zero exit-code propagation from run-tests.ps1 through the pwsh shell step.
  • Did not trigger an actual workflow run — that needs a real PR against the self-hosted runner, which this PR itself will do once opened. Worth watching the Actions tab after this PR is created to confirm GH-RUNNER-FLAUI actually picks up the job and has the toolchain it needs (pwsh itself is assumed present per the spec — the workflow bootstraps the Pester module but not the interpreter).

🤖 Generated with Claude Code

Adds .github/workflows/test.yml: runs the non-integration Pester suite
(./tests/run-tests.ps1) on every pull request, using the self-hosted
Windows runner already connected to this repo (GH-RUNNER-FLAUI) but
previously unused by any workflow. Bootstraps the Pester 5+ module if
missing (idempotent), fails the job on any test failure (no swallowed
exit codes), and caps at timeout-minutes: 20 - there is exactly one
connected runner, so an unbounded hang would block every subsequent
workflow run indefinitely.

Does NOT add a required status check. While blueprinting this, running
the full suite directly on a real Windows dev machine reproduced a real
failure + hang in switch_to_main's "Should fail when there are
uncommitted changes" test (apparent cross-file state pollution across
the 10 test files) - gating merges now would risk blocking everything
on a suite that isn't provably stable. That diagnosis is tracked
separately (bead gitconfig-cm6).

- Added: .github/workflows/test.yml
- Modified: CHANGELOG.md, STATUS.md
- Dependencies: none new; Pester module bootstrap runs on the runner at
  job time
- Tested: forge generator/evaluator loop (spec:
  specs/archive/add-pester-ci-workflow.md) - all 9 acceptance criteria
  (C1-C9, C-final) passed round 1, including the evaluator executing the
  Pester-bootstrap step's script directly in real pwsh for both the
  "already installed" and "missing" branches, and confirming exit-code
  propagation empirically. Did not trigger an actual workflow run (needs
  a real PR against the self-hosted runner) - verified via YAML parse
  (yaml.safe_load) and manual trace instead.

Fixes #221

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@J-MaFf J-MaFf added the enhancement New feature or request label Aug 26, 2026
@J-MaFf J-MaFf self-assigned this Aug 26, 2026
@J-MaFf

J-MaFf commented Aug 26, 2026

Copy link
Copy Markdown
Owner Author

Self-review

Re-read the full diff (.github/workflows/test.yml, CHANGELOG.md, STATUS.md, specs/archive/add-pester-ci-workflow.md) after opening this PR.

  • on: is bare pull_request — no push, no workflow_dispatch, no types: restriction, matching the spec's "pull_request only" requirement exactly.
  • runs-on: [self-hosted, Windows, X64] matches the connected runner's labels exactly (verified via gh api repos/J-MaFf/gitconfig/actions/runners).
  • timeout-minutes: 20 is set at the job level — deliberately chosen given the reproduced full-suite hang, so a stuck run can't block the only connected runner indefinitely.
  • The Pester-bootstrap step is idempotent: traced both branches by hand (module present → logs and continues; module absent → installs then continues) — neither branch fails the job on its own.
  • ./tests/run-tests.ps1 runs with zero arguments (no -IncludeIntegration) under shell: pwsh; no continue-on-error, output suppression, or exit-code swallowing anywhere in the file — a real test failure will fail the job.
  • .github/workflows/claude.yml is untouched, and the branch ruleset (gh api repos/J-MaFf/gitconfig/rulesets/11141898) still shows only non_fast_forward + pull_request — no required status check was added, per the explicit decision not to gate merges until the suite's flakiness is diagnosed.

One caveat worth flagging directly rather than papering over: I have not seen this workflow actually execute on GH-RUNNER-FLAUI — that can only happen once a real PR triggers it, which this PR itself will do. Worth checking the Actions tab after this opens to confirm the runner picks up the job and that pwsh is actually present on it (the workflow bootstraps the Pester module but assumes pwsh itself is already installed, per the spec's explicit scope decision).

No other issues found; nothing to fix in a follow-up commit beyond watching that first real run.

… -PassThru

Discovered by this PR's own new CI workflow: the very first real run on
GH-RUNNER-FLAUI showed 61 genuine Pester failures but the job reported
success.

Root cause: Run.PassThru in the Pester configuration was wired to the
script's own -PassThru switch (default $false), but Invoke-Pester only
returns a results object when Run.PassThru is $true. Called without
-PassThru - the script's own primary documented usage
(".\run-tests.ps1") - $results was $null, so
`$results.FailedCount -gt 0` silently evaluated to $false regardless of
actual failures, and the script always exit 0'd.

Run.PassThru is now always $true internally so the exit-code check has
real data; the caller-facing -PassThru switch's contract (an unrelated,
separately pre-existing gap - the script never actually emitted
$results to its own caller either way) is unaffected by this change.

- Modified: tests/run-tests.ps1
- Tested: parses cleanly (0 syntax errors via
  [Parser]::ParseFile); a scoped zero-match run confirms exit 0 is still
  correct for a clean pass; a deliberately-failing single test confirms
  exit 1 is now correctly produced (previously would have been 0); this
  PR's own re-triggered CI workflow run against the real 61 pre-existing
  failures is the authoritative end-to-end confirmation

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@J-MaFf

J-MaFf commented Aug 26, 2026

Copy link
Copy Markdown
Owner Author

Follow-up: found and fixed a real bug via this PR's own first CI run

The first real trigger of this workflow on GH-RUNNER-FLAUI exposed something the spec's acceptance criteria (and my own self-review) didn't catch: 61 genuine Pester test failures, but the job reported success.

Root cause was in tests/run-tests.ps1 itself (pre-existing, unrelated to this PR's own new code): Invoke-Pester -Configuration $config only returns a results object when Run.PassThru is $true, but that was wired to the script's own -PassThru switch (default $false) rather than always being requested internally. Called plain — .\run-tests.ps1, the script's own primary documented usage, and exactly how this new workflow calls it — $results was $null, so $results.FailedCount -gt 0 silently evaluated to $false no matter how many tests failed, and the script always exit 0'd.

Fixed in 1db18cf: Run.PassThru is now always $true internally for the exit-code check, independent of the caller-facing -PassThru switch.

Verified on the real runner, not just locally: re-triggered the workflow after the fix — same 61 pre-existing failures, but the job now correctly shows completed / failure (previously completed / success). Run: https://github.com/J-MaFf/gitconfig/actions/runs/33003617096

Net effect: this PR's checks tab will show red, correctly — those 61 failures are real and pre-existing (unrelated to this PR's own changes), and are the subject of the separately-tracked flakiness investigation (bead gitconfig-cm6). That's expected and is the workflow doing its job correctly for the first time, not a regression from this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CI workflow to run the Pester test suite on the self-hosted Windows runner

1 participant