Skip to content

feat: add QEMU, Goss and native Molecule testing - #8

Merged
neilime merged 1 commit into
mainfrom
feat/direct-qemu-vm
Nov 4, 2026
Merged

neilime merged 1 commit into
mainfrom
feat/direct-qemu-vm

Conversation

@neilime

@neilime neilime commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Add direct QEMU creation and standardize infrastructure assertions on Goss, with native Molecule scenarios for Ansible deployment, idempotence and reboot checks. Workflows compose focused create/test/destroy actions and retain control of VM lifetime.

  • vm/create selects Lima or QEMU and requires consumer configuration. Both backends expose the same VM and SSH references, with startup rollback and verified teardown.
  • vm/test opens fresh guest logins so provisioning changes to groups take effect. It requires a Goss suite, installs the checksum-pinned verifier, rejects empty suites, and publishes native JUnit, summaries, annotations, PR comments and artifacts after failures. Remove generic command/script execution, runner mode, report globs and Codecov.
  • ansible/test-molecule runs native consumer scenarios without requiring a VM. Keep collection sanity/playbook testing and linting separate.
  • Keep action scripts local, share only required safety contracts, and use dedicated AAA workflows plus focused action-local unit tests. Generate references through the pinned Docker documentation tool.

BREAKING CHANGE: require vm/test.goss-file; remove its generic execution/reporting inputs and test-action outputs; remove unused Ansible overrides and explicit GitHub token inputs. Consumers are being updated together in workstation-manager #308, digital-household #7 and engineering-office #8.

Validation: all 43 local unit tests and repository lint pass; documentation regenerated. Hosted CI passes with both Lima and QEMU, including group changes after SSH connection setup, failed/empty/timed-out suites, native reports and cleanup.

@github-actions

Copy link
Copy Markdown

Hi, thank you for creating your PR, we will check it out very soon

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Ansible tests - playbook - success

Ansible tests - playbook - success

Test Results

Status Count
✅ Passed 2
❌ Failed 0
⏭️ Skipped 0
Total 2

✅ Pass Rate: 100.0%

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Ansible tests - playbook - failure

Ansible tests - playbook - failure

Test Results

Status Count
✅ Passed 1
❌ Failed 1
⏭️ Skipped 0
Total 2

❌ Pass Rate: 50.0%

Failed Tests
  • [fixture] Exercise a synthetic Ansible test assertion: Assert the selected fixture succeeds (suite: valid)
    • File: tests/fixture/ansible/valid.yml:8
    • Error: Task failed: Action failed: Assertion failed

@github-actions

Copy link
Copy Markdown

Ansible tests - sanity - failure

Ansible tests - sanity - failure

Test Results

Status Count
✅ Passed 6
❌ Failed 1
⏭️ Skipped 27
Total 34

❌ Pass Rate: 17.6%

Failed Tests
  • yamllint (suite: ansible-test)
    • File: sanity
    • Error: The test `ansible-test sanity --test yamllint` failed with 2 errors:

@github-actions

Copy link
Copy Markdown

QEMU guest reports

QEMU guest reports

Test Results

Status Count
✅ Passed 1
❌ Failed 0
⏭️ Skipped 0
Total 1

✅ Pass Rate: 100.0%

Coverage

Metric Covered Total Percentage
Lines 1 1 100.00%

Overall: 100.00% 🟢
🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

QEMU runner reports

QEMU runner reports

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • expected-failure (suite: Unknown Suite)
    • Error: Synthetic failure

@github-actions

Copy link
Copy Markdown

Ansible tests - sanity - success

Ansible tests - sanity - success

Test Results

Status Count
✅ Passed 7
❌ Failed 0
⏭️ Skipped 27
Total 34

✅ Pass Rate: 20.6%

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Super-linter summary

Language Validation result
BIOME_FORMAT Pass ✅
BIOME_LINT Pass ✅
CHECKOV Pass ✅
GITHUB_ACTIONS Pass ✅
GITHUB_ACTIONS_ZIZMOR Pass ✅
GITLEAKS Pass ✅
GIT_COMMITLINT Pass ✅
GIT_MERGE_CONFLICT_MARKERS Pass ✅
JSCPD Pass ✅
MARKDOWN Pass ✅
MARKDOWN_PRETTIER Pass ✅
NATURAL_LANGUAGE Pass ✅
PRE_COMMIT Pass ✅
RENOVATE Pass ✅
SPELL_CODESPELL Pass ✅
TRIVY Pass ✅
YAML Pass ✅
YAML_PRETTIER Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter
Super-linter revision: 2da136927bd4a73596db63044b504547c62cb854
Super-linter version: 9.0.0

@github-actions

Copy link
Copy Markdown

VM tests - success - first

VM tests - success - first

Test Results

Status Count
✅ Passed 1
❌ Failed 0
⏭️ Skipped 0
Total 1

✅ Pass Rate: 100.0%

@github-actions

Copy link
Copy Markdown

VM tests - success - repeated

VM tests - success - repeated

Test Results

Status Count
✅ Passed 1
❌ Failed 0
⏭️ Skipped 0
Total 1

✅ Pass Rate: 100.0%

Coverage

Metric Covered Total Percentage
Lines 1 1 100.00%

Overall: 100.00% 🟢
🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

@github-actions

Copy link
Copy Markdown

VM tests - assertion-failure - first

VM tests - assertion-failure - first

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • deliberate-failure (suite: guest-fixture)
    • File: ./tests/fixture/fail.sh
    • Error: Synthetic assertion failure

@github-actions

Copy link
Copy Markdown

VM tests - assertion-failure - repeated

VM tests - assertion-failure - repeated

Test Results

Status Count
✅ Passed 1
❌ Failed 0
⏭️ Skipped 0
Total 1

✅ Pass Rate: 100.0%

Coverage

Metric Covered Total Percentage
Lines 1 1 100.00%

Overall: 100.00% 🟢
🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

@neilime
neilime marked this pull request as ready for review October 31, 2026 11:34
@github-actions

Copy link
Copy Markdown

Goss - __test-actions-qemu.yml - valid - 0

Goss - __test-actions-qemu.yml - valid - 0

Test Results

Status Count
✅ Passed 2
❌ Failed 0
⏭️ Skipped 0
Total 2

✅ Pass Rate: 100.0%

@github-actions

Copy link
Copy Markdown

Goss - __test-actions-qemu.yml - invalid - 0

Goss - __test-actions-qemu.yml - invalid - 0

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • File /iac-missing-synthetic-file exists (suite: goss)
    • Error: Test failed

@github-actions

Copy link
Copy Markdown

Goss - __test-actions-qemu.yml - empty - 0

Goss - __test-actions-qemu.yml - empty - 0

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • consumer-command (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Molecule - success

Molecule - success

Test Results

Status Count
✅ Passed 10
❌ Failed 0
⏭️ Skipped 0
Total 10

✅ Pass Rate: 100.0%

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Molecule - assertion-failure

Molecule - assertion-failure

Test Results

Status Count
✅ Passed 3
❌ Failed 2
⏭️ Skipped 0
Total 5

❌ Pass Rate: 60.0%

Failed Tests
  • scenario (suite: molecule)
    • Error: Molecule failed, timed out or produced no native assertions
  • [smoke] Verify the disposable test configuration: Assert the expected content (suite: verify)
    • File: tests/fixture/molecule/smoke/verify.yml:10
    • Error: Task failed: Action failed: Assertion failed

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Molecule - idempotence-failure

Molecule - idempotence-failure

Test Results

Status Count
✅ Passed 5
❌ Failed 1
⏭️ Skipped 0
Total 6

❌ Pass Rate: 83.3%

Failed Tests
  • scenario (suite: molecule)
    • Error: Molecule failed, timed out or produced no native assertions

@github-actions

Copy link
Copy Markdown

Lima state - success

Lima state - success

Test Results

Status Count
✅ Passed 2
❌ Failed 0
⏭️ Skipped 0
Total 2

✅ Pass Rate: 100.0%

@github-actions

Copy link
Copy Markdown

Lima state - assertion-failure

Lima state - assertion-failure

Test Results

Status Count
✅ Passed 2
❌ Failed 0
⏭️ Skipped 0
Total 2

✅ Pass Rate: 100.0%

@neilime neilime changed the title feat(vm): support direct QEMU and runner-side tests feat(vm): add direct QEMU and opinionated Goss and Molecule tests Oct 31, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Molecule local scenario - success

Molecule local scenario - success

Test Results

Status Count
✅ Passed 7
❌ Failed 0
⏭️ Skipped 1
Total 8

✅ Pass Rate: 87.5%

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Molecule local scenario - failure

Molecule local scenario - failure

Test Results

Status Count
✅ Passed 4
❌ Failed 2
⏭️ Skipped 1
Total 7

❌ Pass Rate: 57.1%

Failed Tests
  • [local-fixture] Verify the local test resource: Assert configuration and selected outcome (suite: assertions)
    • File: tests/fixture/molecule/local/assertions.yml:14
    • Error: Task failed: Action failed: Assertion failed
  • scenario (suite: molecule)
    • Error: Molecule failed, timed out or produced no native assertions

@neilime neilime changed the title feat(vm): add direct QEMU and opinionated Goss and Molecule tests feat: add direct QEMU, Goss and native Molecule actions Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Molecule local scenario - timeout

Molecule local scenario - timeout

Test Results

Status Count
✅ Passed 3
❌ Failed 1
⏭️ Skipped 0
Total 4

❌ Pass Rate: 75.0%

Failed Tests
  • scenario (suite: molecule)
    • Error: Molecule failed, timed out or produced no native assertions

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Goss - vm/test - qemu - valid

Goss - vm/test - qemu - valid

Test Results

Status Count
✅ Passed 2
❌ Failed 0
⏭️ Skipped 0
Total 2

✅ Pass Rate: 100.0%

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Goss - vm/test - qemu - invalid

Goss - vm/test - qemu - invalid

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • File /iac-missing-synthetic-file exists (suite: goss)
    • Error: Test failed

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Goss - vm/test - qemu - empty

Goss - vm/test - qemu - empty

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • consumer-command (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs

@neilime neilime changed the title feat: add direct QEMU, Goss and native Molecule actions feat!: add focused infrastructure testing actions Nov 4, 2026
@neilime
neilime force-pushed the feat/direct-qemu-vm branch 2 times, most recently from 3f6dd9f to 6bd5dbe Compare November 4, 2026 13:48
@neilime neilime changed the title feat!: add focused infrastructure testing actions feat!: add QEMU, Goss and native Molecule testing Nov 4, 2026
@neilime
neilime force-pushed the feat/direct-qemu-vm branch from 6bd5dbe to 32f581a Compare November 4, 2026 15:55
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Goss - qemu - valid

Goss - qemu - valid

Test Results

Status Count
✅ Passed 7
❌ Failed 0
⏭️ Skipped 0
Total 7

✅ Pass Rate: 100.0%

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Goss - qemu - invalid

Goss - qemu - invalid

Test Results

Status Count
✅ Passed 0
❌ Failed 2
⏭️ Skipped 0
Total 2

❌ Pass Rate: 0.0%

Failed Tests
  • goss-execution (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs
  • File /iac-missing-synthetic-file exists (suite: goss)
    • Error: Test failed

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Goss - qemu - empty

Goss - qemu - empty

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • goss-execution (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Goss - qemu - timeout

Goss - qemu - timeout

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • goss-execution (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Goss - lima - valid

Goss - lima - valid

Test Results

Status Count
✅ Passed 7
❌ Failed 0
⏭️ Skipped 0
Total 7

✅ Pass Rate: 100.0%

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Goss - lima - invalid

Goss - lima - invalid

Test Results

Status Count
✅ Passed 0
❌ Failed 2
⏭️ Skipped 0
Total 2

❌ Pass Rate: 0.0%

Failed Tests
  • goss-execution (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs
  • File /iac-missing-synthetic-file exists (suite: goss)
    • Error: Test failed

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Goss - lima - empty

Goss - lima - empty

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • goss-execution (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Goss - lima - timeout

Goss - lima - timeout

Test Results

Status Count
✅ Passed 0
❌ Failed 1
⏭️ Skipped 0
Total 1

❌ Pass Rate: 0.0%

Failed Tests
  • goss-execution (suite: disposable-vm)
    • Error: Guest assertions or report collection failed; inspect test outputs

@neilime
neilime force-pushed the feat/direct-qemu-vm branch from 32f581a to 0d2ce23 Compare November 4, 2026 16:26
@neilime neilime changed the title feat!: add QEMU, Goss and native Molecule testing feat: add QEMU, Goss and native Molecule testing Nov 4, 2026
BREAKING CHANGE: VM assertions require a Goss suite; generic commands, runner execution, coverage inputs and unused action outputs are removed.
@neilime
neilime force-pushed the feat/direct-qemu-vm branch from 0d2ce23 to 5a51a44 Compare November 4, 2026 17:19
@neilime
neilime merged commit 89e3bf9 into main Nov 4, 2026
45 checks passed
@neilime
neilime deleted the feat/direct-qemu-vm branch November 4, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant