GitHub Actions: per-test results from the JUnit reports, and shorter action references

.github/actions/README.md

@@ -1,9 +1,16 @@
 # pluginval GitHub Actions
 
 Two composite actions for using pluginval from another repository's workflows.
+Their `action.yml` files are in `validate/` and `acceptance/` at the repository
+root, which keeps the references short; the scripts they run are in this directory.
 Both run on Linux, macOS and Windows runners, download the matching pluginval
-release (or use a binary you give them), run everything they are given, write a
-results table to the job summary and fail the step if anything failed.
+release (or use a binary you give them), run everything they are given and fail
+the step if anything failed.
+
+Results are shown in two places: the job summary gets a table with a row per
+plugin / config and its test count, followed by the name and message of every
+failed test; and each failed plugin / config gets an error annotation naming its
+failed tests, which also appears in the pull request's checks.
 
 On Linux the plugin is run under `xvfb-run` when there is no `DISPLAY` (install
 `xvfb` on runners that don't have it).
@@ -13,7 +20,7 @@ On Linux the plugin is run under `xvfb-run` when there is no `DISPLAY` (install
 Validates one or more plugins.
 
 ```yaml
-- uses: Tracktion/pluginval/.github/actions/validate@v2
+- uses: Tracktion/pluginval/validate@v2
   with:
     plugins: |
       build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
@@ -27,6 +34,7 @@ Validates one or more plugins.
 | `plugins` | (required) | The plugins to validate, one per line. Each is a path, a glob or an AU id. A glob that matches nothing fails. |
 | `strictness-level` | `5` | The strictness level (1-10). |
 | `args` | | Extra options for `pluginval validate`, split on whitespace. |
+| `report-dir` | | Directory to write the JUnit XML reports to (a temporary directory if empty). |
 | `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
 | `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
 
@@ -41,7 +49,7 @@ Runs the [acceptance tests](../../docs/Acceptance%20testing.md) for every config
 plugin.
 
 ```yaml
-- uses: Tracktion/pluginval/.github/actions/acceptance@v2
+- uses: Tracktion/pluginval/acceptance@v2
   with:
     config-dir: tests/acceptance
     plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
@@ -53,6 +61,7 @@ plugin.
 | `plugin` | | Plugin path or AU id to test every config against, instead of the `plugin` in each config. |
 | `record-missing` | `false` | Record a missing reference and pass, instead of failing. |
 | `timeout-ms` | | Fail a test if it takes longer than this (pluginval's default if empty). |
+| `report-dir` | | Directory to write the JUnit XML reports to (a temporary directory if empty). |
 | `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. |
 | `pluginval-path` | | An existing pluginval binary to use instead of downloading one. |
 
@@ -78,7 +87,7 @@ Both actions set:
 To keep the reports and diffs from a failed run:
 
 ```yaml
-- uses: Tracktion/pluginval/.github/actions/acceptance@v2
+- uses: Tracktion/pluginval/acceptance@v2
   id: acceptance
   with:
     config-dir: tests/acceptance
@@ -90,3 +99,36 @@ To keep the reports and diffs from a failed run:
     name: pluginval-acceptance-${{ runner.os }}
     path: ${{ steps.acceptance.outputs.report-dir }}
 ```
+
+### Per-test results as a pull request check
+
+The actions report failures through the job summary and annotations, which need
+no extra permissions. If you would rather have every pluginval test listed as its
+own check on the pull request, feed the JUnit reports to a reporter action such
+as [mikepenz/action-junit-report](https://github.com/mikepenz/action-junit-report)
+or [dorny/test-reporter](https://github.com/dorny/test-reporter):
+
+```yaml
+jobs:
+  validate:
+    runs-on: macos-latest
+    permissions:
+      contents: read
+      checks: write
+    steps:
+      # ... build the plugin ...
+
+      - uses: Tracktion/pluginval/validate@v2
+        with:
+          plugins: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3
+          report-dir: pluginval-reports
+
+      - uses: mikepenz/action-junit-report@v5
+        if: always()
+        with:
+          report_paths: pluginval-reports/*.xml
+          check_name: pluginval
+```
+
+This needs the `checks: write` permission, which workflows triggered by pull
+requests from forks don't get, so on those the reporter step can't publish.

.github/actions/acceptance/acceptance.sh

@@ -27,6 +27,7 @@ passed=0
 failed=0
 index=0
 rows=""
+details=""
 
 # Every *.json is a config, except the <reference>.wav.json manifests that
 # pluginval writes next to each reference.
@@ -44,22 +45,29 @@ for config in ${configs[@]+"${configs[@]}"}; do
     # pluginval only runs the first entry of an array config, so reject them
     # rather than silently skipping tests.
     if [ "$(tr -d '[:space:]' < "$config" | head -c 1)" = "[" ]; then
-        echo "Array configs aren't supported: use one config per file."
+        failure="Array configs aren't supported: use one config per file."
+        echo "$failure"
         code=1
     else
         run_pluginval test ${options[@]+"${options[@]}"} --junit-xml "$report" "$config"
         code=$?
+        failure="pluginval exited with code $code (see the log)"
     fi
 
     echo "::endgroup::"
 
+    # The per-test results come from the JUnit report; pass / fail from the exit code.
+    read_report "$report"
+
     if [ $code -eq 0 ]; then
         passed=$((passed + 1))
-        rows="$rows| \`$name\` | :white_check_mark: passed |"$'\n'
+        rows="$rows| \`$name\` | :white_check_mark: passed | $REPORT_TESTS |"$'\n'
     else
         failed=$((failed + 1))
-        rows="$rows| \`$name\` | :x: failed |"$'\n'
-        echo "::error::pluginval acceptance test failed: $name"
+        rows="$rows| \`$name\` | :x: failed | $REPORT_TESTS |"$'\n'
+        [ -z "$REPORT_DETAILS" ] && REPORT_DETAILS="  - $failure"$'\n'
+        details="$details- \`$name\`"$'\n'"$REPORT_DETAILS"
+        annotate_error "pluginval acceptance test failed: $name" "${REPORT_NAMES:-$failure}"
     fi
 done
 
@@ -78,9 +86,16 @@ done
 {
     echo "### pluginval acceptance tests"
     echo
-    echo "| Config | Result |"
-    echo "| --- | --- |"
+    echo "| Config | Result | Tests |"
+    echo "| --- | --- | --- |"
     printf '%s' "$rows"
+
+    if [ -n "$details" ]; then
+        echo
+        echo "#### Failed tests"
+        echo
+        printf '%s' "$details"
+    fi
 } >> "$GITHUB_STEP_SUMMARY"
 
 {

.github/actions/common/common.sh

@@ -2,6 +2,8 @@
 # Shared helpers for the pluginval composite actions (sourced, not run).
 # Kept compatible with bash 3.2, which is what macOS runners ship.
 
+COMMON_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
 # Defaults so the scripts can also be run outside of GitHub Actions.
 if [ -z "${RUNNER_OS:-}" ]; then
     if [ "$(uname -s)" = "Darwin" ]; then RUNNER_OS=macOS
@@ -72,3 +74,34 @@ sanitise()
 {
     printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_'
 }
+
+# Reads a JUnit XML report written by pluginval and sets:
+#   REPORT_TESTS    a summary-table cell, e.g. "17" or "2 of 17 failed"
+#   REPORT_NAMES    "name: message" per failed test, "; " separated (for the annotation)
+#   REPORT_DETAILS  a markdown list item per failed test (for the job summary)
+# All three are empty if there is no report.
+read_report()
+{
+    REPORT_TESTS=""
+    REPORT_NAMES=""
+    REPORT_DETAILS=""
+
+    [ -f "$1" ] || return 0
+
+    local kind first second
+    while IFS=$'\t' read -r kind first second; do
+        if [ "$kind" = "T" ]; then
+            if [ "$second" -gt 0 ]; then REPORT_TESTS="$second of $first failed"; else REPORT_TESTS="$first"; fi
+        elif [ "$kind" = "F" ]; then
+            REPORT_NAMES="${REPORT_NAMES:+$REPORT_NAMES; }$first: $second"
+            REPORT_DETAILS="$REPORT_DETAILS  - **${first//|/\\|}**: ${second//|/\\|}"$'\n'
+        fi
+    done < <(awk -f "$COMMON_DIR/junit.awk" "$1")
+}
+
+# Prints an ::error annotation, escaping the message as workflow commands need.
+annotate_error()
+{
+    local message="${2//%/%25}"
+    echo "::error title=$1::$message"
+}

.github/actions/common/junit.awk

@@ -0,0 +1,42 @@
+# Summarises a pluginval JUnit XML report (--junit-xml) as tab-separated lines:
+#   T <tests> <failed>          the totals (failures + errors)
+#   F <test name> <message>     one per failed test
+# Written for POSIX awk (macOS ships BSD awk), so no gawk extensions.
+
+function attr(s, key,    start, rest)
+{
+    start = index(s, " " key "=\"")
+    if (start == 0) return ""
+    rest = substr(s, start + length(key) + 3)
+    return substr(rest, 1, index(rest, "\"") - 1)
+}
+
+function unescape(s)
+{
+    gsub(/&#1[03];/, " ", s)
+    gsub(/&lt;/, "<", s)
+    gsub(/&gt;/, ">", s)
+    gsub(/&quot;/, "\"", s)
+    gsub(/&apos;/, "\047", s)
+    gsub(/&amp;/, "\\&", s)
+    gsub(/[\t ]+/, " ", s)
+    return s
+}
+
+{ sub(/\r$/, ""); xml = xml " " $0 }
+
+END {
+    count = split(xml, cases, "<testcase ")
+    printf "T\t%d\t%d\n", attr(cases[1], "tests"), attr(cases[1], "failures") + attr(cases[1], "errors")
+
+    for (i = 2; i <= count; ++i)
+    {
+        problem = index(cases[i], "<failure")
+        if (problem == 0) problem = index(cases[i], "<error")
+        if (problem == 0) continue
+
+        message = unescape(attr(substr(cases[i], problem), "message"))
+        if (length(message) > 300) message = substr(message, 1, 300) "..."
+        printf "F\t%s\t%s\n", unescape(attr(" " cases[i], "name")), message
+    }
+}

.github/actions/validate/validate.sh

@@ -14,6 +14,7 @@ passed=0
 failed=0
 index=0
 rows=""
+details=""
 
 # One plugin per line (not comma-separated: AU ids contain commas). Globs are
 # left to expand; anything that matches nothing is passed through and fails.
@@ -33,14 +34,20 @@ for plugin in ${plugins[@]+"${plugins[@]}"}; do
     run_pluginval validate --strictness-level "${INPUT_STRICTNESS_LEVEL:-5}" --junit-xml "$report" ${INPUT_ARGS:-} "$plugin"
     code=$?
     echo "::endgroup::"
+    failure="pluginval exited with code $code (see the log)"
+
+    # The per-test results come from the JUnit report; pass / fail from the exit code.
+    read_report "$report"
 
     if [ $code -eq 0 ]; then
         passed=$((passed + 1))
-        rows="$rows| \`$plugin\` | :white_check_mark: passed |"$'\n'
+        rows="$rows| \`$plugin\` | :white_check_mark: passed | $REPORT_TESTS |"$'\n'
     else
         failed=$((failed + 1))
-        rows="$rows| \`$plugin\` | :x: failed |"$'\n'
-        echo "::error::pluginval validation failed: $plugin"
+        rows="$rows| \`$plugin\` | :x: failed | $REPORT_TESTS |"$'\n'
+        [ -z "$REPORT_DETAILS" ] && REPORT_DETAILS="  - $failure"$'\n'
+        details="$details- \`$plugin\`"$'\n'"$REPORT_DETAILS"
+        annotate_error "pluginval validation failed: $plugin" "${REPORT_NAMES:-$failure}"
     fi
 done
 
@@ -52,9 +59,16 @@ fi
 {
     echo "### pluginval validate"
     echo
-    echo "| Plugin | Result |"
-    echo "| --- | --- |"
+    echo "| Plugin | Result | Tests |"
+    echo "| --- | --- | --- |"
     printf '%s' "$rows"
+
+    if [ -n "$details" ]; then
+        echo
+        echo "#### Failed tests"
+        echo
+        printf '%s' "$details"
+    fi
 } >> "$GITHUB_STEP_SUMMARY"
 
 {

.github/workflows/acceptance.yml

@@ -69,7 +69,7 @@ jobs:
         working-directory: build
         run: ctest -C Release --output-on-failure -R pluginval.acceptance
 
-      # The reusable actions in .github/actions, run against the build above
+      # The reusable actions (validate/ and acceptance/), run against the build above
       # (pluginval-path) rather than a downloaded release.
       - name: Prepare action fixtures
         id: fixtures
@@ -100,7 +100,7 @@ jobs:
 
       - name: "Action: validate"
         id: validate
-        uses: ./.github/actions/validate
+        uses: ./validate
         with:
           pluginval-path: ${{ steps.fixtures.outputs.pluginval }}
           args: --skip-gui-tests
@@ -109,7 +109,7 @@ jobs:
       - name: "Action: validate (missing plugin, expected to fail)"
         id: validate-missing
         continue-on-error: true
-        uses: ./.github/actions/validate
+        uses: ./validate
         with:
           pluginval-path: ${{ steps.fixtures.outputs.pluginval }}
           args: --skip-gui-tests
@@ -119,7 +119,7 @@ jobs:
 
       - name: "Action: acceptance"
         id: acceptance
-        uses: ./.github/actions/acceptance
+        uses: ./acceptance
         with:
           pluginval-path: ${{ steps.fixtures.outputs.pluginval }}
           config-dir: build/action-fixtures/good
@@ -128,7 +128,7 @@ jobs:
       - name: "Action: acceptance (bad configs, expected to fail)"
         id: acceptance-bad
         continue-on-error: true
-        uses: ./.github/actions/acceptance
+        uses: ./acceptance
         with:
           pluginval-path: ${{ steps.fixtures.outputs.pluginval }}
           config-dir: build/action-fixtures/bad
@@ -137,15 +137,15 @@ jobs:
       - name: "Action: acceptance (missing reference, expected to fail)"
         id: acceptance-missing
         continue-on-error: true
-        uses: ./.github/actions/acceptance
+        uses: ./acceptance
         with:
           pluginval-path: ${{ steps.fixtures.outputs.pluginval }}
           config-dir: build/action-fixtures/missing
           plugin: build/tests/test_plugins/tone_generator/pluginval_tone_generator_artefacts/Release/VST3/pluginval Tone Generator.vst3
 
       - name: "Action: acceptance (record the missing reference)"
         id: acceptance-record
-        uses: ./.github/actions/acceptance
+        uses: ./acceptance
         with:
           pluginval-path: ${{ steps.fixtures.outputs.pluginval }}
           config-dir: build/action-fixtures/missing

CHANGELIST.md

@@ -6,7 +6,7 @@
 - Added `--junit-xml <file>` and `--json-results <file>` to both `pluginval validate` and `pluginval test` to write machine-readable reports for CI (also settable as `JUNIT_XML` / `JSON_RESULTS`). A report that can't be written fails the run. See `docs/Adding pluginval to CI.md`
 - Added acceptance testing: `pluginval test <config.json>` renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting `0` on a match and `1` on a mismatch (with a diff WAV). Use `--record-missing` to record new references. See `docs/Acceptance testing.md`
 - Added `--plugin <plugin>` to `pluginval test` to run a config against a given plugin instead of the one it names (the config's `plugin` can then be omitted)
-- Added two GitHub Actions for use from other repositories: `Tracktion/pluginval/.github/actions/validate` (validate a list of plugins) and `.../acceptance` (run every acceptance config in a directory against a built plugin). See `.github/actions/README.md`
+- Added two GitHub Actions for use from other repositories: `Tracktion/pluginval/validate` (validate a list of plugins) and `Tracktion/pluginval/acceptance` (run every acceptance config in a directory against a built plugin). See `.github/actions/README.md`
 - Restructured the command line into subcommands: `pluginval validate [options] <plugin>` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate`
 - **Deprecated:** the flat flags `--validate <plugin>`, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval <plugin>` remains a silent shorthand for `validate`
 - Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline