Skip to content

ci: extract test.yml assertions into standalone scripts - #38

Merged
locus313 merged 2 commits into
mainfrom
ci/extract-test-scripts
Aug 28, 2026
Merged

ci: extract test.yml assertions into standalone scripts#38
locus313 merged 2 commits into
mainfrom
ci/extract-test-scripts

Conversation

@locus313

@locus313 locus313 commented Aug 28, 2026

Copy link
Copy Markdown
Owner

Summary

Reviewed the CI testing setup. test.yml embedded every test case as inline bash directly in the workflow YAML — adding a single new test case meant editing the workflow file itself, and the ~360-line YAML mixed fixture provisioning (users, packages) with test logic (assertions, grep checks).

This splits test logic out into two scripts that mirror the existing test.sh pattern (same run_test harness style):

  • .github/scripts/functional-tests.sh — checks previously in the test job (config validation, invalid method handling, empty user array, function presence, etc.), plus a new self-update round-trip test
  • .github/scripts/integration-tests.sh — checks previously in the integration-test job (real sync run, permission checks, both symlink-attack tests), plus a new file ownership check

test.yml now only provisions CI fixtures (packages, temporary Linux test users) and calls the scripts. Adding a new test case is now just a test_* function + run_test line in the relevant script — no workflow YAML changes required.

Existing behavior is unchanged: same test users, same fixtures, same assertions and pass/fail messages.

New test coverage

While reviewing what else was worth adding:

  • test_self_update_round_trip — runs a full --self-update against the real latest GitHub release on a scratch copy of the script. The sha256 digest verification added in security: verify self-update checksum and stop leaking GitHub token via argv #37 previously had zero behavioral coverage; the old check only grepped for function names, not that verification actually works.
  • test_file_ownership_correct — verifies .ssh/authorized_keys are chowned to the target user. Only permission bits (700/600) were checked before; ownership is a separate code path (get_user_gid/chown) that could silently leave files root-owned.

Considered and skipped as too flaky/heavy for the value:

  • Asserting the GitHub token never appears in ps output — timing-dependent.
  • A negative self-update test (tampered digest → rejected) — would require mocking curl/GitHub API responses.

Testing

  • bash -n on both new scripts, shellcheck -S warning (matches CI's lint severity) — clean
  • Ran functional-tests.sh locally end-to-end against real test users, the live github.com/locus313.keys endpoint, and the live self-update round trip — all 11 pass
  • Ran integration-tests.sh locally end-to-end, including both symlink-attack tests against real /root/.ssh canaries and the new ownership check — all 5 pass
  • bash test.sh — all 10 existing unit tests still pass

Bug found during extraction

The scripts originally had set -o pipefail. Several tests intentionally run the sync script expecting it to fail, then pipe its output into grep -q "expected error message". With pipefail, the pipeline's exit status was the sync script's nonzero code rather than grep's (matching) result, which flipped those assertions to fail even when the correct error was printed. Fixed by dropping pipefail (each test_* function's own return code is what run_test checks, so it isn't needed).

Docs updated

AGENTS.md, .github/copilot-instructions.md (maintenance matrix + CI/CD section), and TESTING.md now point at the new script locations instead of test.yml.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

locus313 and others added 2 commits August 27, 2026 18:58
test.yml previously embedded every test case as inline bash in the
workflow YAML, so adding a test meant editing the workflow. Split the
assertions into .github/scripts/functional-tests.sh and
integration-tests.sh (using the same run_test-harness pattern as
test.sh); the workflow now only provisions CI fixtures (packages, test
users) and calls the scripts. New tests are added as a test_* function
+ run_test line in the appropriate script, no workflow changes needed.

Behavior is unchanged — same test users, same fixtures, same pass/fail
messages, verified by running both scripts locally end-to-end
(including the symlink-attack integration tests) against a live
GitHub API/keys endpoint.

Also fixes a `set -o pipefail` bug introduced during extraction: piping
the (intentionally failing) sync script into `grep -q` made pipefail
report the script's exit code instead of grep's match result, flipping
an assertion that checks for an expected error message.

Updates AGENTS.md, .github/copilot-instructions.md, and TESTING.md
references to point at the new script locations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two gaps found while reviewing CI coverage:

- Self-update's sha256 digest verification (added in #37) had zero test
  coverage — the existing check only grepped for function names, not
  behavior. test_self_update_round_trip runs a full --self-update against
  the real latest GitHub release on a scratch copy of the script, so a
  broken digest extraction or verification would fail CI.
- Integration tests checked authorized_keys/.ssh permission bits (700/600)
  but never verified chown ownership, even though get_user_gid/chown is
  its own code path that could silently leave files root-owned.
  test_file_ownership_correct checks both are owned by the target user.

Verified locally end-to-end: 11/11 functional tests and 5/5 integration
tests pass, including the new round-trip and ownership checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@locus313
locus313 merged commit 426bc6b into main Aug 28, 2026
7 checks passed
@locus313
locus313 deleted the ci/extract-test-scripts branch August 28, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant