Introduce conformance test CLI #346
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| name: Conformance | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| branches: [main] | ||
| merge_group: | ||
| schedule: | ||
| - cron: '17 3 * * *' | ||
| workflow_dispatch: | ||
| inputs: | ||
| scope: | ||
| type: choice | ||
| options: [pr, full] | ||
| default: pr | ||
| update_check: | ||
| type: boolean | ||
| default: false | ||
|
|
||
| concurrency: | ||
| # A schedule event resolves github.ref to the default branch, so without the | ||
| # event split the nightly full run and a main push share one group and a | ||
| # second push cancels the queued nightly. | ||
| group: ${{ github.workflow }}-${{ github.event_name == 'schedule' && 'nightly' || github.ref }} | ||
| cancel-in-progress: ${{ github.event_name == 'pull_request' }} | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| env: | ||
| CONF_SCOPE: ${{ (github.event_name == 'schedule' || inputs.scope == 'full') && 'full' || 'pr' }} | ||
| CONF_REQUIRE: 1 | ||
|
|
||
| jobs: | ||
| discover: | ||
| runs-on: ubuntu-24.04 | ||
| outputs: | ||
| suites: ${{ steps.suites.outputs.names }} | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - id: suites | ||
| run: | | ||
| names=$(python3 scripts/conformance suites --format json | | ||
| python3 -c 'import json,sys; print(json.dumps(json.load(sys.stdin)["suites"]))') | ||
| echo "names=$names" >> "$GITHUB_OUTPUT" | ||
|
|
||
| harness: | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - run: python3 scripts/conformance selftest | ||
|
|
||
| payload: | ||
| needs: discover | ||
| if: needs.discover.outputs.suites != '[]' | ||
| runs-on: ubuntu-24.04-arm | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - run: make conformance-payloads | ||
| - run: tar -C externals -cf conformance-payloads.tar payloads | ||
| - uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: conformance-payloads | ||
| path: conformance-payloads.tar | ||
| if-no-files-found: error | ||
|
|
||
| qemu: | ||
| needs: [discover, payload] | ||
| if: needs.discover.outputs.suites != '[]' | ||
| runs-on: [self-hosted, macOS, ARM64] | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - uses: actions/download-artifact@v8 | ||
| with: | ||
| name: conformance-payloads | ||
| - run: mkdir -p externals && tar -C externals -xf conformance-payloads.tar | ||
| - run: bash tests/fetch-fixtures.sh | ||
| - run: make test-conformance BACKEND=qemu | ||
|
|
||
| elfuse: | ||
| needs: [discover, payload, qemu] | ||
| if: needs.discover.outputs.suites != '[]' | ||
| runs-on: [self-hosted, macOS, ARM64] | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - uses: actions/download-artifact@v8 | ||
| with: | ||
| name: conformance-payloads | ||
| - run: mkdir -p externals && tar -C externals -xf conformance-payloads.tar | ||
| - run: bash tests/fetch-fixtures.sh | ||
| - run: make elfuse | ||
| - run: make test-conformance BACKEND=elfuse | ||
|
|
||
| conformance: | ||
| name: Conformance (make test-conformance) | ||
| needs: [discover, harness, payload, qemu, elfuse] | ||
| if: always() | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - env: | ||
| SUITES: ${{ needs.discover.outputs.suites }} | ||
| DISCOVER: ${{ needs.discover.result }} | ||
| HARNESS: ${{ needs.harness.result }} | ||
| PAYLOAD: ${{ needs.payload.result }} | ||
| QEMU: ${{ needs.qemu.result }} | ||
| ELFUSE: ${{ needs.elfuse.result }} | ||
| run: | | ||
| [ "$DISCOVER" = success ] | ||
| [ "$HARNESS" = success ] | ||
| if [ "$SUITES" = '[]' ]; then | ||
| [ "$PAYLOAD $QEMU $ELFUSE" = 'skipped skipped skipped' ] | ||
| else | ||
| [ "$PAYLOAD $QEMU $ELFUSE" = 'success success success' ] | ||
| fi | ||
|
|
||
| update-check: | ||
| if: github.event_name == 'workflow_dispatch' && inputs.update_check | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - run: python3 scripts/conformance pins check | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # Conformance Harness | ||
|
|
||
| The harness runs registered Linux test suites on elfuse and a QEMU reference. | ||
| It records suite status separately from the expectation verdict. The command | ||
| reference is in [testing.md](testing.md#conformance-tests). | ||
|
|
||
| ## Results | ||
|
|
||
| `run` writes `results.json` below `<results>/<suite>/<backend>/<stamp>-<pid>/`. | ||
| That file is the canonical artifact: `schema_version: 1`, `kind: run`, run | ||
| metadata, derived counts and gate, and case records. Loading rejects a gate or | ||
| count that disagrees with the cases. An empty run is red. | ||
|
|
||
| Each attempt records `normal`, `timeout`, `signal`, or `transport`, elapsed | ||
| microseconds, output paths, and an exit code or signal when applicable. Case | ||
| statuses are `PASS`, `FAIL`, `SKIP`, `CONF`, `WARN`, `BROK`, `TIMEOUT`, | ||
| `CRASH`, `INCONSISTENT`, and `ERROR`. Verdicts are `as_expected`, | ||
| `unexpected_failure`, `unexpected_pass`, `flaked`, `filtered`, and `error`. | ||
|
|
||
| JSON list output also has `schema_version: 1` and a `kind` field. Requested | ||
| machine data uses stdout. Diagnostics use stderr. | ||
|
|
||
| Exit codes are: | ||
|
|
||
| - `0`: the operation succeeded or the run is green. | ||
| - `1`: a completed run or artifact check is red. | ||
| - `2`: the command, configuration, or operation is invalid. | ||
| - `3`: a non-writing pin or selection check found drift. | ||
| - `77`: an optional prerequisite is absent. `--require` and `CONF_REQUIRE=1` | ||
| promote it to `2`. | ||
|
|
||
| ## IDs and Selection | ||
|
|
||
| Case IDs have one of these forms: | ||
|
|
||
| ```text | ||
| <suite>:<group> | ||
| <suite>:<group>/<case>[/<parameter>...] | ||
| ``` | ||
|
|
||
| Selectors and expectation matchers use shell globs across the complete ID. | ||
| A bare group selector also selects its cases. An unmatched selector is an | ||
|
Contributor
There was a problem hiding this comment. True for So a matcher naming a bare group silently covers nothing, and only a full-scope run's |
||
| error. | ||
|
|
||
| A selection file assigns each upstream launch group to `pr`, `full`, or a | ||
| declined group with a reason. PR groups run in both scopes. Enabled entries | ||
| may set `timeout_s` and suite-specific case filters. | ||
|
|
||
| ## Expectations | ||
|
|
||
| Expectation files are JSONC and accept comments and trailing commas. A suite | ||
| has a base file, one leaf per backend, and optional `flaky.jsonc`. Files contain | ||
| ordered actions; the last matching non-quarantine action wins. The first | ||
| effective action is `expect_pass` for `*`. | ||
|
|
||
| Actions are `expect_pass`, `expect_failure`, `expect_conf`, `skip`, and | ||
| `quarantine`. Every non-pass action needs a reason. `quarantine` is valid only | ||
| in `flaky.jsonc`; it runs the case alone for at most three attempts and reports | ||
| test mismatches as `flaked`. Harness errors remain red. A full run rejects | ||
| matchers that select no case. | ||
|
|
||
| A skipped expectation prevents launch. `--bootstrap` launches skipped cases | ||
| and records status without applying expectations. `expectations seed` derives | ||
| actions from bootstrap statuses or red verdicts. It refuses harness errors. | ||
|
|
||
| ## Payloads and Pins | ||
|
|
||
| Payloads live below `externals/payloads/` and are not committed. A fingerprint | ||
| hashes the pin and builder inputs. `manifest.json` records the fingerprint and | ||
| each staged file or symlink. Verification detects missing, extra, changed, and | ||
| stale content before a run starts. | ||
|
|
||
| Pins are schema-checked JSON. `pins update` validates the new pin before | ||
| replacing the file. | ||
|
|
||
| ## Suite Interface | ||
|
|
||
| `tests/conformance/providers/__init__.py` is the static suite registry; | ||
| `Provider` in `providers/base.py` declares what a suite supplies. | ||
|
|
||
| The shared runner owns expectation loading, skip handling, unresolved batch | ||
| reruns, quarantine retries, result ordering, and judgment. Providers map | ||
| suite output to statuses. Backends return process invocations. A provider | ||
| translates host paths through `backend.guest_path()` before putting them in | ||
| argv; `Backend.run` forwards argv unchanged, because only the provider knows | ||
| which elements are paths. QEMU records non-timeout shell statuses as exit | ||
| codes. Providers interpret `128+n` through the suite contract because the | ||
| shell cannot distinguish it from a plain exit with the same value. | ||
|
|
||
| The elfuse backend starts one `build/elfuse --timeout 0` process for each | ||
| command. The QEMU backend starts one VM through `tests/qemu-runner.sh`, shares | ||
| the repository read-only at `/mnt/host`, and executes commands over SSH. | ||
|
|
||
| ## Make and CI | ||
|
|
||
| The Make targets take their suite list from the registry through | ||
| `scripts/conformance suites`. An empty registry makes suite targets print | ||
| `SKIP`; harness selftests still run. | ||
|
|
||
| `.github/workflows/conformance.yml` runs QEMU before elfuse and gates on the | ||
| required `Conformance (make test-conformance)` job. Pull requests use the PR | ||
| scope. Schedules and `scope=full` dispatches use the full scope. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| .PHONY: test-conformance-harness test-conformance test-conformance-full \ | ||
| conformance-payloads clean-payloads update-pins | ||
|
|
||
| CONFORMANCE := python3 scripts/conformance | ||
| # The suite registry lives in tests/conformance/providers/__init__.py. On a | ||
| # failed discovery the marker fails every consumer instead of skipping. | ||
| CONF_SUITES ?= $(shell $(CONFORMANCE) suites || echo suite-discovery-failed) | ||
| BACKEND ?= elfuse | ||
| TEST ?= | ||
| CONF_JOBS ?= 4 | ||
| CONF_RESULTS ?= $(BUILD_DIR)/conformance | ||
| CONF_RUN = $(CONFORMANCE) run | ||
| CONF_SCOPE ?= pr | ||
| CONF_SELECT = $(if $(TEST),$(foreach id,$(TEST),--case '$(id)'),--scope $(CONF_SCOPE)) | ||
| CONF_NO_SUITES = $(if $(CONF_SUITES),,@printf "$(YELLOW)SKIP$(RESET) no conformance suites registered\n") | ||
| # foreach inserts spaces, but RUN_OPTIONAL_SKIP77 expands as a recipe line. | ||
| define conf-newline | ||
|
|
||
|
|
||
| endef | ||
| define conf-lane | ||
| $(foreach s,$(CONF_SUITES),$(call RUN_OPTIONAL_SKIP77,$(CONF_RUN) $(s) $(1) --backend $(BACKEND) --jobs $(CONF_JOBS) --results $(CONF_RESULTS),test-$(s)$(2))$(conf-newline)) | ||
| endef | ||
|
|
||
| ## Run the conformance harness selftests (hermetic) | ||
| test-conformance-harness: | ||
| @$(CONFORMANCE) selftest | ||
|
|
||
| ## Run every suite's CONF_SCOPE subset, or TEST=ID... (BACKEND=elfuse|qemu|all) | ||
| test-conformance: | ||
| $(CONF_NO_SUITES) | ||
| $(call conf-lane,$(CONF_SELECT),) | ||
|
|
||
| ## Run every suite in full, the nightly shape | ||
| test-conformance-full: | ||
| $(CONF_NO_SUITES) | ||
| $(call conf-lane,--scope full,-full) | ||
|
|
||
| ## Build every conformance payload under externals/payloads/ | ||
| conformance-payloads: | ||
| $(CONF_NO_SUITES) | ||
| $(foreach s,$(CONF_SUITES),$(CONFORMANCE) payload build $(s) &&) true | ||
|
|
||
| ## Remove the conformance payloads (they survive clean and distclean) | ||
| clean-payloads: | ||
| rm -rf externals/payloads | ||
|
|
||
| UPDATE_CHECK ?= | ||
|
|
||
| ## Refresh the conformance pins from upstream (UPDATE_CHECK=1 to report only) | ||
| update-pins: | ||
| $(CONF_NO_SUITES) | ||
| $(foreach s,$(CONF_SUITES),$(CONFORMANCE) pins $(if $(filter 1,$(UPDATE_CHECK)),check,update) $(s) $(if $(CONF_REF_$(s)),--ref $(CONF_REF_$(s))) &&) true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The registry is empty on this branch, so
suitesreturns[], the payload/qemu/elfuse jobs all skip, and this branch asserts that skip as success. The required check is green while nothing runs, and it stays green if the last provider is ever dropped from the registry.Once the first suite lands this needs a floor: assert the discovered list is non-empty, or compare it against a checked-in expected inventory.