[matthewgrossman]

Summary

This PR makes it possible to run a testkit test on top of tmachine. A small runner builds a test family's tests for the guest, boots one tmachine environment with OpenShell installed, runs the tests inside it, and checks the JUnit results. On CI, an advisory lane runs one smoke test on Ubuntu with rootful Docker.

nix run .#openshell-test-runner -- \
  --environment ubuntu-docker-rootful --installer deb \
  --family conformance --package sandbox

Most of the runner and the tmachine Session API come from @elezar's draft #4629.

Stack

1. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773 (this PR): the tmachine Session API, a runner for one family in one tmachine environment, a smoke suite, and an advisory CI lane with an explicit matrix.
2. test(suites): port the universal conformance suites onto testkit and run them from the plan #4774: ports the conformance suites that every driver must pass to testkit and runs them in more environments. It is being reshaped to build on this PR without plans.
3. Next: the features and drivers suites move to testkit.
4. Later: plans (which suites run where), once features show what they need.

Deferred from this PR

• Plans, the environment registry, the source catalog, matrix generation and strict mode, plus optional test IDs and testkit-spec. They are kept on a branch at be103d6 (the previous head of this PR) and come back after features. Until then, CI lists what runs where as an explicit matrix in e2e-suites.yml.
• More environments and suites. One environment and one smoke test run here.
• Gating. The e2e-suites lane is advisory and not part of the Core E2E result.
• arm64. CI has no arm64 runner with KVM yet.
• Runner robustness: a hung SSH or QMP connection, and a signal that arrives while the runner is busy with blocking work.

Dropped from #4629, not planned:

• requires, capabilities and waivers. A test that fails in an environment fails the lane. #[ignore = "<issue link>"] takes a broken test out everywhere.
• The target file. Tests find the gateway the way the CLI does, through OPENSHELL_GATEWAY and OPENSHELL_BIN.

Out of scope for the whole stack: version compatibility guarantees between the conformance suite and gateways, and running tests from other repositories.

Related Issue

Part of #3954

Changes

• tmachine: the Session API; SSH, scp, and exec timeouts; down() never panics; QEMU is cleaned up on SIGTERM; the SSH port is TMACHINE_SSH_PORT. An openshell-test-runner flake package. Environment setup is unchanged.
• tests/runner (openshell-test-runner): picks the tmachine environment and installer by name, builds the family's packages (a workspace under tests/suites/) as a nextest archive for the guest's musl target, waits for the gateway with testkit-ready, runs nextest in the guest, and writes junit.xml, nextest.log and, after a failure, the guest's diagnostics to --results. A run passes if nextest succeeds and its report shows at least one test and no failures. --filter narrows with a nextest filterset, and --keep keeps the VM for debugging.
• tests/suites/conformance/sandbox: the smoke suite (create a sandbox, exec echo, delete it). mise run e2e:testkit runs it against a local Docker gateway.
• .github/workflows/e2e-suites.yml (reusable), with an explicit include matrix of one row, and an e2e-suites job in Branch E2E (label test:e2e). JUnit and logs are uploaded as e2e-results-<family>-<package>-<environment>-<installer>. It is not part of the Core E2E result yet.
• The lockfile alignment check, enrolling the testkit and conformance workspaces. fmt, clippy, and nextest for the runner, tmachine, and suite in Branch Checks.
• TESTING.md, AGENTS.md, CONTRIBUTING.md and the testkit and suite READMEs describe the runner.

Testing

• Checks appropriate to the affected code and behavior pass
  • Locally: fmt and clippy -D warnings for testkit, the runner, tmachine and the conformance suite; the runner (10), tmachine (8) and testkit unit tests; the lockfile alignment check and its tests; the SPDX header check. Every commit builds with --locked.
  • nix run .#openshell-test-runner -- --help builds and runs.
  • The runner checks JUnit parsing against a real report from the nextest version the guest installs.
  • Not yet run on this head: a VM run (the amd64 guest can't boot on an arm64 Mac) and Branch E2E. The plans-based version of this lane passed Branch E2E on be103d6 (run).
• Unit tests added/updated
• E2E tests added/updated

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (TESTING.md, AGENTS.md, CONTRIBUTING.md, tests/testkit/README.md, tests/suites/conformance/sandbox/README.md)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4773.docs.buildwithfern.com/openshell

[drew]

The source catalog does not account for conditional attributes, so a healthy lane can fail JUnit verification. For example:

/// Disabled on every target.
#[cfg(any())]
#[testkit::test]
async fn disabled(cx: &mut Context) -> testkit::Result { /* ... */ }

I reproduced the catalog including disabled and result verification reporting it missing, although Cargo correctly excludes it. Platform-specific #[cfg(...)] tests have the same problem, and #[cfg_attr(..., ignore)] is not recognized by the literal #[ignore] check either.

Could we resolve these attributes for the guest build, or explicitly reject unsupported conditional declarations with a clear error, rather than adding them to the expected test set? Reproduced on 1da9c3b832bdae01be7f4022bd9e030a84b2dbe4; the relevant catalog and parser behavior remains present at 4781a500c1f8e4064c1c64e8b37c963fa4a46902.

[matthewgrossman]

Thanks, good catch. Rather than evaluate cfg for the guest target, testkit-spec now refuses it: a #[cfg] or #[cfg_attr] on a #[testkit::test] function, or a file-level #![cfg] in a file that has tests, is an error from both the macro and the catalog scan, so the catalog and the build can't disagree. A test that only applies to some environments belongs in a features or drivers suite, and #[ignore = "<issue>"] remains the way to skip one everywhere. Tests are in testkit-spec (rejects_conditional_tests, scan_rejects_conditional_files_with_tests), and tests/testkit/README.md documents the rule. Fixed in 1b80fad.

[github-actions]

Label test:e2e applied for f244394. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.