Repository navigation
test(suites): port the universal conformance suites onto testkit and run them from the plan - #4774
Draft
matthewgrossman wants to merge 16 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
matthewgrossman
added this pull request to stack #4776
October 20, 2026 23:38
|
🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell |
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
2 times, most recently
from
October 21, 2026 20:40
6d64703 to
e3d9f01
Compare
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 21, 2026 20:49
e3d9f01 to
5e2cbc5
Compare
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 21, 2026 23:45
5e2cbc5 to
cda205f
Compare
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 22, 2026 04:51
cda205f to
7c75fbf
Compare
matthewgrossman
removed this pull request from stack #4776
October 22, 2026 04:51
matthewgrossman
added this pull request to stack #4824
October 22, 2026 04:51
5 tasks done
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 22, 2026 05:29
7c75fbf to
bb39723
Compare
matthewgrossman
changed the title
matthewgrossman
removed this pull request from stack #4824
October 22, 2026 05:29
matthewgrossman
added this pull request to stack #4825
October 22, 2026 05:30
matthewgrossman
removed this pull request from stack #4825
October 22, 2026 16:01
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 22, 2026 16:01
bb39723 to
1cb1dc8
Compare
matthewgrossman
added this pull request to stack #4836
October 22, 2026 16:01
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 22, 2026 16:37
1cb1dc8 to
fbcd389
Compare
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 22, 2026 17:31
986fdcf to
2fdcd59
Compare
2 tasks done
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 23, 2026 00:01
2fdcd59 to
33b8dc1
Compare
matthewgrossman
changed the title
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 23, 2026 03:37
33b8dc1 to
db85da9
Compare
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 23, 2026 04:00
db85da9 to
6f5c621
Compare
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 23, 2026 04:46
6f5c621 to
b9e6bfe
Compare
`#[testkit::test]` takes an optional `id = "t-0123abcd"` argument that gives a test a stable identity surviving renames and moves. The attribute's parsing moves into a new testkit-spec crate, which the test runner's catalog also uses to list tests from source without compiling them. The scanner refuses testkit tests inside inline modules. `#[should_panic]` stays rejected. There are no per-environment requirements: a test that cannot pass everywhere belongs in another family, and a broken one is `#[ignore]`d with an issue link. The testkit lockfile is also realigned with the root lockfile. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add a library API that boots an environment, runs commands in it with timeouts, copies files in and out, collects diagnostics and tears it down, so a caller can drive a machine without writing an Ansible playbook per suite. SSH and exec timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from leaving QEMU running. Add the `test-runner` flake package, which runs tests/runner with the toolchains and VM tools it needs. The Session API and test-runner package are adapted from #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…like the root `cargo --locked` only checks a lockfile against its own manifest. A workspace outside the root that builds root crates through path dependencies, such as the testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two lockfiles can resolve a shared crate to different versions without any check failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link against the AWS-LC that Nix provides. check_cargo_lockfile_alignment.py fails when an aligned workspace locks a registry crate that the root lockfile also contains at a version the root lockfile does not have. With --fix it re-locks the workspace starting from the root lockfile. tests/testkit is the first aligned workspace. The script and its tests come from #4629 (a37ebe9). Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add the `sandbox` crate to the conformance workspace with one testkit test that creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone. It is deliberately small: it exists so a plan can run a real test through the runner on CI before the existing suites move to testkit. The workspace's other crate, the CLI conformance suite, keeps running as it does today. The conformance workspace joins the lockfile alignment check, so its lockfile is realigned with the root lockfile. The test is adapted from the `sandbox-lifecycle` suite in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
`openshell-test-runner` expands a plan (tests/plans/*.yaml) into one job per environment, builds the family's nextest archive for the guest, boots the environment through the tmachine Session API, runs nextest in it and brings back the JUnit report and logs. No per-suite Ansible playbook is involved. The runner points the tests at the system under test with environment variables: OPENSHELL_GATEWAY (the gateway the installer registered, from the environment's `gateway` field) and OPENSHELL_BIN (the installed CLI). Nextest runs with `--test-threads 1` and a slow-timeout that terminates a hung test. The JUnit report is checked against the catalog, so a missing or skipped expected test fails the job; `#[ignore]` tests are not expected and nextest's skipped case for them is accepted. `--strict` fails on an unsupported environment, a family with no tests, or an empty selection. `--emit-github-matrix` prints the GitHub Actions matrix for a plan, so the plan is the only list of what runs where. One environment is defined, amd64 Ubuntu with rootful Docker and the Debian package, which is the one CI can provision today; tests/plans/cpu-tests.yaml runs the conformance family on it. Further environments (Podman, k3s, arm64) are added with the suites that need them. There are no waivers or per-test requirements: the plan picks suites, and a broken test is `#[ignore]`d. Most of this comes from #4629, adapted to testkit finding the gateway the way the CLI does instead of reading a target file. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and clippy for the new `sandbox` suite crate, wherever tests/testkit already runs them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they run through `mise run e2e:testkit` against a local Docker gateway and, on CI, through the test runner rather than the unit-test job. The lockfile task and the Cargo lockfiles job also run the alignment check, and its own tests run with `mise run test`. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add a reusable workflow that takes a plan. A planning job runs `openshell-test-runner test --plan <plan> --emit-github-matrix`, and one job per environment, on the runner label the matrix gives, runs `nix run .#test-runner -- test --plan <plan> --env <id> --strict`. The jobs use the same integration inputs as the existing tmachine lanes, cache the tmachine disks keyed on the Ansible tree, upload the JUnit reports and logs whether or not the run passes, and an aggregator job reports the overall result. Branch E2E calls it for tests/plans/cpu-tests.yaml when the test:e2e label is set. The job is not in the Core E2E result's needs yet, so it starts out advisory. The existing conformance, feature and driver lanes are unchanged. Modeled on the E2E Suites workflow in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Describe how a plan runs on CI and locally in TESTING.md and a new tests/plans/README.md, add a README for the sandbox suite, point the testkit README at the runner, and add rows for tests/suites, tests/tmachine, tests/runner, tests/plans and tests/environments to the project-structure tables in AGENTS.md and CONTRIBUTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
`mise run e2e:testkit` runs the conformance sandbox suite against a Docker-backed gateway started by e2e/with-docker-gateway.sh, the same way the runner runs it in a guest. It is the quick way to run the suite locally. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ependency Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add cx.track and cx.adopt, so a test that creates a sandbox through the CLI still cleans it up after a failure: track the name before the command, adopt it afterwards for an SDK handle. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ent-workspace suites to testkit Add the lifecycle, cli, persistent_workspace and file_transfer test files to the sandbox crate. The lifecycle, persistent-workspace and file-transfer tests are adapted from #4629. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start, and the one that can't yet (MXC) isn't conformant until it can. The policy-advisor suite needs no port: it is a filter over the existing conformance tests that no workflow runs. The OCI image, Keycloak and Podman userns suites are ported separately, in the features and drivers families. The old conformance/cli crate stays until the release lanes move to plans. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The file-transfer suite builds a Git repository on the machine that runs the tests, and the old conformance playbook installed Git itself. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The k3s installer now waits until the Agent Sandbox controller reconciles a gated probe Sandbox, so a test that starts right after boot doesn't race it. The check moves out of the old conformance playbook into tests/tmachine/ansible/playbooks/k3s-controller-ready.yaml, which the installer and the old playbook both include. Tests themselves don't wait for the controller. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
A plan can't run part of the conformance family in one environment and a different part in another, or conformance would mean different things in different places. The runner refuses such a plan and names the environments that differ. A test that can't pass everywhere belongs in another family. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add the Podman (rootful and rootless, RPM) and k3s and HA k3s environments and put them, with Docker, in the cpu-tests plan, which now runs the whole conformance family (smoke, lifecycle, cli, persistent-workspace and file-transfer) on all five. The tmachine disk cache in e2e-suites.yml is keyed per environment; the k3s environments skip it as integration-runner.yml does. The old suites and lanes still run unchanged. arm64 is not defined. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
force-pushed
the
test/3954-migrate-suites-to-testkit/matthewgrossman
branch
from
October 23, 2026 05:30
b9e6bfe to
919f767
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the universal conformance suites (CLI, lifecycle, file-transfer, persistent-workspace) onto testkit and runs them from the
cpu-testsplan on Docker, Podman (rootful and rootless) and k3s (single and HA). This is PR 2 of the plan inconformance-plan.md, stacked on the runner.It adds the new suites without removing anything: the old suites, the hand-written release lanes (
conformance-integration) and the old crates are untouched. Switchingconformance-integrationto the runner, and deleting the old CLI suite, is PR 3, after each environment has been shown green here a few times.The lifecycle, persistent-workspace and file-transfer ports start from @elezar's draft #4629.
Stack
Part of a stack; each PR targets the one below it. The tmachine move (#4726) and the testkit library (#4724) have merged.
SessionAPI and a smoke suite running on CI.Still to come: (3) ship
openshell-conformanceas one program withtestkit::main(), point the host lanes at it, delete the old CLI suite and switchconformance-integrationto the runner; (4) migratefeaturesanddrivers(oci-image, Keycloak providers, Podman userns, e2e-podman) and deletecrates/openshell-conformance.Related Issue
Part of #3954
Changes
tests/suites/conformance/sandbox: test fileslifecycle,cli,persistent_workspaceandfile_transfernext tosmoke.persistent_workspace_survives_stop_startstays in conformance: every conformant driver keeps the workspace across stop and start (MXC isn't conformant until it can restart a stopped sandbox).cx.trackandcx.adoptin testkit, for sandboxes a test creates through the CLI.tests/environments: adds Podman (Fedora 44, rootful and rootless, RPM) and k3s and HA k3s (Ubuntu 24.04), amd64 only.tests/plans/cpu-tests.yaml: the wholeconformancefamily on all five environments.conformanceentry must select the same suites in every environment, or the runner refuses the plan and names the environments that differ.k3s-controller-ready.yaml, moved out of the old conformance playbook, which includes it too). Tests don't wait for it.e2e-suites.yml: the tmachine disk cache is keyed per environment. The k3s environments skip the cache, asintegration-runner.ymldoes; its history doesn't say why, so that is still to find out.test-tools.yaml), for the.gitignorefiltering test.Old test to new test
smoke(scenario: status, create, get Ready, list, exec, delete, list empty)sandbox/tests/cli.rs::cli_create_exec_list_deletesmoke.rs::create_exec_deleteis the SDK variant (in #4773).sandbox_lifecyclestop_start_preserves_workspacepersistent_workspace.rs::persistent_workspace_survives_stop_start(SDK) andcli.rs::cli_stop_start_delete(CLI)lifecycle.rs::restart_stopped_sandbox.sandbox_lifecyclestopped_can_be_deletedlifecycle.rs::stopped_sandbox_can_be_deletedcli_stop_start_delete.file_transfer::round_tripfile_transfer.rs::upload_download_round_tripfile_transfer::git_filteringfile_transfer.rs::upload_respects_gitignoregitwhere the tests run.file_transfer::path_safetyfile_transfer.rs::download_stays_inside_workspaceNew with no old counterpart (from #4629):
lifecycle.rs::long_running_exec_delete,run_to_completion_exit_delete,default_command_sandbox_is_listed_ready.policy_advisorhas no port: it is a filter (binary(policy_advisor)) over existing conformance tests that no workflow runs, so it goes with the old suite.Not in this PR
conformance-integration: PR 3 per the plan, after each environment is shown green several times viaworkflow_dispatch(fallback: switch Docker first).oci-image, Keycloak providers, Podman userns,e2e-podman: PR 4, infeaturesanddrivers. Earlier drafts of this PR ported them; they were removed because the runner has no way yet for a VM to hand a suite extra inputs (the plan's decision 8).integration-runner.ymlskips caching k3s before caching it here.Testing
workflow_dispatchon this head is running; results will be posted here. The ports passed on Docker on an earlier head (21 tests); Podman and k3s have not run through the runner yet.Checklist
🤖 Generated with Claude Code