Add --plugin to pluginval test and GitHub Actions for validate and acceptance tests

.github/actions/README.md

@@ -0,0 +1,92 @@
+# pluginval GitHub Actions
+
+Two composite actions for using pluginval from another repository's workflows.
+Both run on Linux, macOS and Windows runners, download the matching pluginval
+release (or use a binary you give them), run everything they are given, write a
+results table to the job summary and fail the step if anything failed.
+
+On Linux the plugin is run under `xvfb-run` when there is no `DISPLAY` (install
+`xvfb` on runners that don't have it).
+
+## validate
+
+Validates one or more plugins.
+
+```yaml
+- uses: Tracktion/pluginval/.github/actions/validate@v2
+  with:
+    plugins: |
+      build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
+      build/MyPlugin_artefacts/Release/AU/MyPlugin.component
+    strictness-level: 10
+    args: --skip-gui-tests
+```
+
+| Input | Default | |
+| --- | --- | --- |
+| `plugins` | (required) | The plugins to validate, one per line. Each is a path, a glob or an AU id. A glob that matches nothing fails. |
+| `strictness-level` | `5` | The strictness level (1-10). |
+| `args` | | Extra options for `pluginval validate`, split on whitespace. |
+| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
+| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
+
+Every `pluginval validate` option can also be set as an environment variable on
+the step (e.g. `env: { TIMEOUT_MS: 60000 }`), which is the way to pass values
+containing spaces.
+
+## acceptance
+
+Runs the [acceptance tests](../../docs/Acceptance%20testing.md) for every config
+(`*.json`) found recursively under a directory, optionally against one built
+plugin.
+
+```yaml
+- uses: Tracktion/pluginval/.github/actions/acceptance@v2
+  with:
+    config-dir: tests/acceptance
+    plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
+```
+
+| Input | Default | |
+| --- | --- | --- |
+| `config-dir` | (required) | Directory searched recursively for configs. |
+| `plugin` | | Plugin path or AU id to test every config against, instead of the `plugin` in each config. |
+| `record-missing` | `false` | Record a missing reference and pass, instead of failing. |
+| `timeout-ms` | | Fail a test if it takes longer than this (pluginval's default if empty). |
+| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
+| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
+
+Notes:
+
+- Every `*.json` under `config-dir` is treated as a config, apart from the
+  `<reference>.wav.json` manifests. Keep other JSON files (and plugin bundles,
+  which contain their own) out of it.
+- A config holding an array of tests is reported as a failure: use one config
+  per file.
+- Relative `plugin` paths resolve against the working directory; the input,
+  state and reference paths in a config resolve against the config's directory.
+
+## Outputs
+
+Both actions set:
+
+| Output | |
+| --- | --- |
+| `passed` / `failed` | How many plugins / configs passed and failed. |
+| `report-dir` | A directory holding a JUnit XML report for each one. For `acceptance` it also has a `diffs` directory with the `*-diff.wav` files found under `config-dir`. |
+
+To keep the reports and diffs from a failed run:
+
+```yaml
+- uses: Tracktion/pluginval/.github/actions/acceptance@v2
+  id: acceptance
+  with:
+    config-dir: tests/acceptance
+    plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
+
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: pluginval-acceptance-${{ runner.os }}
+    path: ${{ steps.acceptance.outputs.report-dir }}
+```

.github/actions/acceptance/acceptance.sh

@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Runs "pluginval test" on every config found (recursively) under
+# INPUT_CONFIG_DIR, runs them all, and fails if any failed. See action.yml for
+# the inputs.
+
+set -u
+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../common/common.sh"
+
+resolve_pluginval
+
+config_dir="${INPUT_CONFIG_DIR:-}"
+
+if [ ! -d "$config_dir" ]; then
+    echo "::error::config-dir is not a directory: $config_dir"
+    exit 1
+fi
+
+report_dir="${INPUT_REPORT_DIR:-$RUNNER_TEMP/pluginval-reports/acceptance}"
+mkdir -p "$report_dir"
+
+options=()
+[ -n "${INPUT_PLUGIN:-}" ] && options+=(--plugin "$INPUT_PLUGIN")
+[ "${INPUT_RECORD_MISSING:-false}" = "true" ] && options+=(--record-missing)
+[ -n "${INPUT_TIMEOUT_MS:-}" ] && options+=(--timeout-ms "$INPUT_TIMEOUT_MS")
+
+passed=0
+failed=0
+index=0
+rows=""
+
+# Every *.json is a config, except the <reference>.wav.json manifests that
+# pluginval writes next to each reference.
+IFS=$'\n'
+configs=( $(find "$config_dir" -type f -name '*.json' ! -name '*.wav.json' | sort) )
+unset IFS
+
+for config in ${configs[@]+"${configs[@]}"}; do
+    index=$((index + 1))
+    name="${config#"$config_dir"/}"
+    report="$report_dir/$index-$(sanitise "$(basename "$config" .json)").xml"
+
+    echo "::group::pluginval test: $name"
+
+    # pluginval only runs the first entry of an array config, so reject them
+    # rather than silently skipping tests.
+    if [ "$(tr -d '[:space:]' < "$config" | head -c 1)" = "[" ]; then
+        echo "Array configs aren't supported: use one config per file."
+        code=1
+    else
+        run_pluginval test ${options[@]+"${options[@]}"} --junit-xml "$report" "$config"
+        code=$?
+    fi
+
+    echo "::endgroup::"
+
+    if [ $code -eq 0 ]; then
+        passed=$((passed + 1))
+        rows="$rows| \`$name\` | :white_check_mark: passed |"$'\n'
+    else
+        failed=$((failed + 1))
+        rows="$rows| \`$name\` | :x: failed |"$'\n'
+        echo "::error::pluginval acceptance test failed: $name"
+    fi
+done
+
+if [ $index -eq 0 ]; then
+    echo "::error::No configs (*.json) found under $config_dir"
+    exit 1
+fi
+
+# Gather the diff WAVs written next to failing references, so they can be
+# uploaded along with the reports.
+find "$config_dir" -type f -name '*-diff.wav' | while IFS= read -r diff; do
+    mkdir -p "$report_dir/diffs"
+    cp "$diff" "$report_dir/diffs/$(sanitise "${diff#"$config_dir"/}")"
+done
+
+{
+    echo "### pluginval acceptance tests"
+    echo
+    echo "| Config | Result |"
+    echo "| --- | --- |"
+    printf '%s' "$rows"
+} >> "$GITHUB_STEP_SUMMARY"
+
+{
+    echo "passed=$passed"
+    echo "failed=$failed"
+    echo "report-dir=$report_dir"
+} >> "$GITHUB_OUTPUT"
+
+echo "pluginval acceptance tests: $passed passed, $failed failed"
+[ $failed -eq 0 ]

.github/actions/acceptance/action.yml

@@ -0,0 +1,51 @@
+name: pluginval acceptance tests
+description: Runs pluginval's golden-file acceptance tests for every config in a directory.
+
+inputs:
+  config-dir:
+    description: >
+      Directory searched recursively for test configs (*.json). The
+      <reference>.wav.json manifests are skipped.
+    required: true
+  plugin:
+    description: >
+      Plugin path or AU id to test every config against, instead of the
+      "plugin" in each config.
+    default: ''
+  record-missing:
+    description: Record a missing reference and pass, instead of failing.
+    default: 'false'
+  timeout-ms:
+    description: Fail a test if it takes longer than this (pluginval's default if empty).
+    default: ''
+  pluginval-version:
+    description: The pluginval release to download - a tag (e.g. v2.0.0-beta1) or "latest".
+    default: v2.0.0-beta1
+  pluginval-path:
+    description: Path to an existing pluginval binary to use instead of downloading one.
+    default: ''
+
+outputs:
+  passed:
+    description: The number of configs that passed.
+    value: ${{ steps.run.outputs.passed }}
+  failed:
+    description: The number of configs that failed.
+    value: ${{ steps.run.outputs.failed }}
+  report-dir:
+    description: Directory holding a JUnit XML report per config, plus diffs/ with any diff WAVs.
+    value: ${{ steps.run.outputs.report-dir }}
+
+runs:
+  using: composite
+  steps:
+    - id: run
+      shell: bash
+      env:
+        INPUT_CONFIG_DIR: ${{ inputs.config-dir }}
+        INPUT_PLUGIN: ${{ inputs.plugin }}
+        INPUT_RECORD_MISSING: ${{ inputs.record-missing }}
+        INPUT_TIMEOUT_MS: ${{ inputs.timeout-ms }}
+        INPUT_PLUGINVAL_VERSION: ${{ inputs.pluginval-version }}
+        INPUT_PLUGINVAL_PATH: ${{ inputs.pluginval-path }}
+      run: bash "$GITHUB_ACTION_PATH/acceptance.sh"

.github/actions/common/common.sh

@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# Shared helpers for the pluginval composite actions (sourced, not run).
+# Kept compatible with bash 3.2, which is what macOS runners ship.
+
+# Defaults so the scripts can also be run outside of GitHub Actions.
+if [ -z "${RUNNER_OS:-}" ]; then
+    if [ "$(uname -s)" = "Darwin" ]; then RUNNER_OS=macOS
+    elif [ "$(uname -s)" = "Linux" ]; then RUNNER_OS=Linux
+    else RUNNER_OS=Windows
+    fi
+fi
+
+: "${RUNNER_TEMP:=$(mktemp -d)}"
+: "${GITHUB_STEP_SUMMARY:=/dev/null}"
+: "${GITHUB_OUTPUT:=/dev/null}"
+
+# Sets PLUGINVAL_BIN to the pluginval binary: INPUT_PLUGINVAL_PATH if given,
+# otherwise the INPUT_PLUGINVAL_VERSION release ("latest" or a tag like v2.0.0-beta1)
+# downloaded for this runner's OS.
+resolve_pluginval()
+{
+    if [ -n "${INPUT_PLUGINVAL_PATH:-}" ]; then
+        PLUGINVAL_BIN="$INPUT_PLUGINVAL_PATH"
+    else
+        local version="${INPUT_PLUGINVAL_VERSION:-latest}"
+        local dir="$RUNNER_TEMP/pluginval-$version"
+        local url="https://github.com/Tracktion/pluginval/releases/download/$version/pluginval_$RUNNER_OS.zip"
+
+        if [ "$version" = "latest" ]; then
+            url="https://github.com/Tracktion/pluginval/releases/latest/download/pluginval_$RUNNER_OS.zip"
+        fi
+
+        case "$RUNNER_OS" in
+            macOS)   PLUGINVAL_BIN="$dir/pluginval.app/Contents/MacOS/pluginval" ;;
+            Windows) PLUGINVAL_BIN="$dir/pluginval.exe" ;;
+            *)       PLUGINVAL_BIN="$dir/pluginval" ;;
+        esac
+
+        if [ ! -f "$PLUGINVAL_BIN" ]; then
+            echo "Downloading $url"
+            mkdir -p "$dir"
+
+            if ! curl -fsSL --retry 3 "$url" -o "$dir/pluginval.zip"; then
+                echo "::error::Couldn't download pluginval $version for $RUNNER_OS ($url)"
+                exit 1
+            fi
+
+            unzip -q -o "$dir/pluginval.zip" -d "$dir"
+            chmod +x "$PLUGINVAL_BIN"
+        fi
+    fi
+
+    if [ ! -f "$PLUGINVAL_BIN" ]; then
+        echo "::error::pluginval binary not found: $PLUGINVAL_BIN"
+        exit 1
+    fi
+}
+
+# Runs pluginval with the given arguments. On Linux without a display it is
+# wrapped in xvfb-run, as plugins generally need an X server to load.
+run_pluginval()
+{
+    if [ "$RUNNER_OS" = "Linux" ] && [ -z "${DISPLAY:-}" ] && command -v xvfb-run > /dev/null; then
+        xvfb-run --auto-servernum "$PLUGINVAL_BIN" "$@"
+    else
+        "$PLUGINVAL_BIN" "$@"
+    fi
+}
+
+# Prints a file-name-safe version of the argument.
+sanitise()
+{
+    printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_'
+}

.github/actions/validate/action.yml

@@ -0,0 +1,48 @@
+name: pluginval validate
+description: Validates one or more audio plugins with pluginval.
+
+inputs:
+  plugins:
+    description: >
+      The plugins to validate, one per line. Each is a path, a glob (e.g.
+      build/*_artefacts/Release/VST3/*.vst3) or an AU id.
+    required: true
+  strictness-level:
+    description: The strictness level (1-10).
+    default: '5'
+  args:
+    description: >
+      Extra options for "pluginval validate", split on whitespace (e.g.
+      "--skip-gui-tests --timeout-ms 60000"). Any option can also be set as an
+      environment variable on the step.
+    default: ''
+  pluginval-version:
+    description: The pluginval release to download - a tag (e.g. v2.0.0-beta1) or "latest".
+    default: v2.0.0-beta1
+  pluginval-path:
+    description: Path to an existing pluginval binary to use instead of downloading one.
+    default: ''
+
+outputs:
+  passed:
+    description: The number of plugins that passed.
+    value: ${{ steps.run.outputs.passed }}
+  failed:
+    description: The number of plugins that failed.
+    value: ${{ steps.run.outputs.failed }}
+  report-dir:
+    description: Directory holding a JUnit XML report per plugin.
+    value: ${{ steps.run.outputs.report-dir }}
+
+runs:
+  using: composite
+  steps:
+    - id: run
+      shell: bash
+      env:
+        INPUT_PLUGINS: ${{ inputs.plugins }}
+        INPUT_STRICTNESS_LEVEL: ${{ inputs.strictness-level }}
+        INPUT_ARGS: ${{ inputs.args }}
+        INPUT_PLUGINVAL_VERSION: ${{ inputs.pluginval-version }}
+        INPUT_PLUGINVAL_PATH: ${{ inputs.pluginval-path }}
+      run: bash "$GITHUB_ACTION_PATH/validate.sh"