Introduce conformance test CLI - #346
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
d07c9e3 to
9482ae7
Compare
jserv
left a comment
jserv
left a comment
There was a problem hiding this comment.
Introduce re-usable shell scripts as common library.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
9482ae7 to
40f41ac
Compare
Extracted the SSH part out. |
1523520 to
417be2a
Compare
qemu-runner.sh and test-matrix.sh each spelled the ssh option list for the test VM, and the copies had drifted: the runner kept the peer alive for 10 s x 6 and the matrix for 15 s x 4. tests/lib/qemu-ssh.sh now holds the one list. qemu_ssh_opts fills QEMU_SSH_OPTS at call time so a caller can still wrap its ssh in timeout(1), which cannot wrap a shell function. Both lanes keep the 60 s dead-peer budget.
The parser owns one capability-first command tree used by Make and CI. Suite registration cannot add commands. run writes results.json before the backend stops, so a failing teardown cannot lose a completed lane. qemu-runner.sh stores VM state across start and stop commands. A command that starts a VM owns its cleanup trap, preserving the trap installed by test-matrix.sh. Start rejects an unusable guest /tmp, and stop verifies the recorded process still names its pidfile before sending a signal. tests/test-qemu-runner-stop.sh pins that check with a recycled pid and with a process whose argv names the pidfile.
docs/testing.md lists every public command and its defaults. The conformance reference defines the result and extension contracts.
417be2a to
f303d4b
Compare
jserv
left a comment
jserv
left a comment
There was a problem hiding this comment.
Rebase latest main branch and resolve conflicts.
| run: | | ||
| [ "$DISCOVER" = success ] | ||
| [ "$HARNESS" = success ] | ||
| if [ "$SUITES" = '[]' ]; then |
There was a problem hiding this comment.
The registry is empty on this branch, so suites returns [], the payload/qemu/elfuse jobs all skip, and this branch asserts that skip as success. The required check is green while nothing runs, and it stays green if the last provider is ever dropped from the registry.
Once the first suite lands this needs a floor: assert the discovered list is non-empty, or compare it against a checked-in expected inventory.
| out: Dict[str, List[str]] = {} | ||
| for (suite, group), members in sorted(by_group.items()): | ||
| actions = {wanted.get(m) for m in members} | ||
| if whole_groups and len(actions) == 1 and None not in actions and len(members) > 1: |
There was a problem hiding this comment.
whole_groups is set for any run whose scope is not cases, so a --scope pr run collapses to suite:group/* even when it covered only part of the group (a pr subset, or an entry with only filters). Two failing cases out of a larger group seed s:g/*.
The untouched siblings then inherit expect_failure, and the next full run reports them as unexpected_pass. Collapse only when the run actually covered the whole group.
| ``` | ||
|
|
||
| Selectors and expectation matchers use shell globs across the complete ID. | ||
| A bare group selector also selects its cases. An unmatched selector is an |
There was a problem hiding this comment.
True for --case selectors, which selection.resolve_ids expands to pattern plus pattern/*, but not for expectation matchers: expectations.resolve calls fnmatch directly, and ids.matches("s:g", "s:g/c") is False.
So a matcher naming a bare group silently covers nothing, and only a full-scope run's stale() reports it. Either split this sentence per consumer, or give resolve the same expansion.
| if lost: | ||
| return Invocation(execution="transport", wall_us=inv.wall_us, | ||
| stdout=str(out_path), stderr=inv.stderr) | ||
| timed_out = rc == 137 and inv.wall_us >= timeout_s * 1_000_000 |
There was a problem hiding this comment.
The tiebreaker compares the guest's exit status against the host's total ssh wall time, which also covers connect, auth and teardown, so it can pass timeout_s while the guest command itself ran for a fraction of that. A case killed by SIGKILL for an unrelated reason (guest OOM, the suite killing its own child) then records TIMEOUT instead of CRASH.
Both statuses are red and both seed skip, so the cost is the recorded status rather than the verdict. An explicit marker printed by the remote script when timeout fires would make it exact.
| try: | ||
| # The lock lives in a shared namespace, so another uid may own it. | ||
| self.lock_file.parent.mkdir(parents=True, exist_ok=True) | ||
| fd = os.open(self.lock_file, os.O_RDWR | os.O_CREAT, 0o600) |
There was a problem hiding this comment.
ElfuseBackend.lock_file is /tmp/elfuse-conformance-<uid>.lock, a predictable name in a shared directory, and this open follows symlinks, so another local user can pre-create it as a symlink and redirect the flock.
os.O_NOFOLLOW closes it. The comment just above already notes the namespace is shared.
2 participants
Build extensible CLI to host gvisor and LTP (and potentially much more) conformance test suites
Summary by cubic
Introduces a conformance test harness that runs registered Linux suites (gvisor, LTP, and more) against elfuse and a QEMU reference, judging results against JSONC expectations. The
scripts/conformanceCLI has a fixed capability-first command set that suite registration cannot extend.New Features
scripts/conformancecommands for listing suites, enumerating cases, running lanes, seeding expectations, and writing reports.Other Changes
/tmpsetup fails, arm its cleanup per CLI command, and verify a recorded pid still belongs to the VM before stopping.tests/lib/qemu-ssh.shfor the runner and test matrix.make checkvia selftests and addsdocs/conformance.mdplus the command reference indocs/testing.md.Written for commit f303d4b. Summary will update on new commits.