From 49bcf55e77d84a919364154a62196ab4947ceb8e Mon Sep 17 00:00:00 2001 From: David Rowland Date: Tue, 13 Oct 2026 12:55:45 +0100 Subject: [PATCH 1/3] Added --junit-xml and --json-results report output to validate and test Both commands can now write machine-readable reports through a shared report_writer. A placeholder error report is written before a run and replaced on completion, and a timeout writes a timed-out report, so a report always agrees with the exit code. A report that can't be written fails the run. --- CHANGELIST.md | 1 + CLAUDE.md | 35 ++++- CMakeLists.txt | 2 + docs/Acceptance testing.md | 7 + docs/Adding pluginval to CI.md | 59 ++++++++ docs/Command line options.md | 4 + source/CommandLine.cpp | 8 + source/CommandLineTests.cpp | 217 +++++++++++++++++++++++++++ source/PluginTests.h | 2 + source/PluginvalSettings.h | 18 ++- source/ReportWriter.cpp | 217 +++++++++++++++++++++++++++ source/ReportWriter.h | 78 ++++++++++ source/SettingsParser.cpp | 19 +++ source/SettingsParser.h | 2 + source/Validator.cpp | 81 +++++++++- source/acceptance/AcceptanceTest.cpp | 51 ++++--- source/acceptance/AcceptanceTest.h | 5 +- source/acceptance/TestReporter.cpp | 40 ++++- source/acceptance/TestReporter.h | 12 +- tests/acceptance/CMakeLists.txt | 6 +- 20 files changed, 836 insertions(+), 28 deletions(-) create mode 100644 source/ReportWriter.cpp create mode 100644 source/ReportWriter.h diff --git a/CHANGELIST.md b/CHANGELIST.md index b53db34..c3f546e 100644 --- a/CHANGELIST.md +++ b/CHANGELIST.md @@ -3,6 +3,7 @@ ### 2.0.0 - Updated JUCE to 9.0.3 - Fixed the crash handler so a backtrace is shown again when a plugin crashes, and made it async-signal-safe [#144] +- Added `--junit-xml ` and `--json-results ` to both `pluginval validate` and `pluginval test` to write machine-readable reports for CI (also settable as `JUNIT_XML` / `JSON_RESULTS`). A report that can't be written fails the run. See `docs/Adding pluginval to CI.md` - Added acceptance testing: `pluginval test ` renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting `0` on a match and `1` on a mismatch (with a diff WAV). Use `--record-missing` to record new references. See `docs/Acceptance testing.md` - Restructured the command line into subcommands: `pluginval validate [options] ` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate` - **Deprecated:** the flat flags `--validate `, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval ` remains a silent shorthand for `validate` diff --git a/CLAUDE.md b/CLAUDE.md index ff21a61..e4a4a20 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -84,6 +84,7 @@ pluginval/ │ ├── PluginvalSettings.h # Unified settings struct + JSON mapping + toPluginTestOptions() │ ├── SettingsParser.cpp/h # CLI/env/config -> merged JSON -> settings; child-process handoff │ ├── SettingsSerializer.cpp/h # JSON load/save + value coercions (comma lists, hex seed) +│ ├── ReportWriter.cpp/h # JUnit XML + JSON report files (shared by validate and test) │ ├── CrashHandler.cpp/h # Crash reporting utilities │ ├── TestUtilities.cpp/h # Helper functions for tests │ ├── RTCheck.h # Real-time safety checking macros @@ -98,7 +99,7 @@ pluginval/ │ │ ├── TestConfig.cpp/h # snake_case JSON config struct (independent of PluginvalSettings) │ │ ├── AcceptanceTest.cpp/h # plugin load + state + input + render; record-or-compare orchestrator │ │ ├── ReferenceComparator.cpp/h # Comparator interface + registry + SampleComparator -│ │ └── TestReporter.cpp/h # text + JSON result, exit code +│ │ └── TestReporter.cpp/h # text + JSON result, exit code, conversion to a report_writer::Report │ └── tests/ # Individual test implementations │ ├── BasicTests.cpp # Core plugin tests (info, state, audio) │ ├── BusTests.cpp # Audio bus configuration tests @@ -316,6 +317,36 @@ spec: `tests/acceptance/Acceptance testing design.md`; end-user guide: multiplexing, automation, extra comparators, child-process isolation) are notes only. +### Report Output (`--junit-xml` / `--json-results`) + +Both `validate` and `test` can write machine-readable report files through one +shared writer, `report_writer` (`source/ReportWriter.cpp/h`). Stdout and the text +log are unaffected. + +- **Model**: a neutral `report_writer::Report` (command, target, properties and a + list of `TestCase`s with a `passed` / `failed` / `error` status, messages, + duration and optional JSON `details`). `toJUnitXml()` builds a single + `` with `juce::XmlElement`; `toJson()` emits snake_case keys with a + `summary` block. `write()` skips unset files and creates parent directories; if it fails, the + run fails (exit `1`). +- **Two adapters**: `toReport()` in `Validator.cpp` converts the + `juce::UnitTestRunner::TestResult` array collected in `runTests()` (one case per + `beginTest`); `acceptance::reporter::toReport()` converts an acceptance + `TestResult` (its existing JSON goes in `details`). +- **Options**: for `validate` they are ordinary settings (`junitXml` / + `jsonResults` in `PluginvalSettings`, resolved to `juce::File`s on + `PluginTests::Options`), so env vars (`JUNIT_XML` / `JSON_RESULTS`), `--config` + and the child-process handoff all work. For `test` they are parsed in + `settings_parser::dispatch()` into `DispatchResult` and carried on + `acceptance::RunOptions`. +- **Incomplete runs**: before a run starts, `makeIncomplete()` writes a + placeholder report with a single `error` case; the real report overwrites it on + completion, so a crash leaves an error report rather than a stale or missing + one. On a timeout the thread that ends the process writes a "timed out" error + report as its last act (`PluginsUnitTestRunner`'s `onTimeout` for validate, the + acceptance `Watchdog` for test), so the report always agrees with the exit code + even if the tests finish in the moment before the process is terminated. + ### Test Framework Tests are self-registering. To find all tests, look for static instances: @@ -467,6 +498,8 @@ Key options (for `validate`): - `--timeout-ms [ms]` - Test timeout (default: 30000, -1 for none) - `--verbose` - Enable verbose logging - `--output-dir [dir]` - Directory for log files +- `--junit-xml [file]` - Write a JUnit XML report (also accepted by `test`) +- `--json-results [file]` - Write a JSON report (also accepted by `test`) - `--sample-rates [list]` - Comma-separated sample rates - `--block-sizes [list]` - Comma-separated block sizes - `--rtcheck [disabled|enabled|relaxed]` - Real-time safety checking diff --git a/CMakeLists.txt b/CMakeLists.txt index e682e9c..d5cd63b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -134,6 +134,7 @@ set(SourceFiles source/MainComponent.h source/PluginTests.h source/PluginvalSettings.h + source/ReportWriter.h source/SettingsParser.h source/SettingsSerializer.h source/TestUtilities.h @@ -151,6 +152,7 @@ set(SourceFiles source/Main.cpp source/MainComponent.cpp source/PluginTests.cpp + source/ReportWriter.cpp source/SettingsParser.cpp source/SettingsSerializer.cpp source/tests/BasicTests.cpp diff --git a/docs/Acceptance testing.md b/docs/Acceptance testing.md index 6352c71..018305f 100644 --- a/docs/Acceptance testing.md +++ b/docs/Acceptance testing.md @@ -89,6 +89,13 @@ Any other option is an error. `RECORD_MISSING=1`. - `--timeout-ms ` - fail (exit `1`) if the run takes longer than this (default `30000`, `-1` to never time out). Also settable as `TIMEOUT_MS`. +- `--junit-xml ` - write a JUnit XML report, for CI systems that display + test results. Also settable as `JUNIT_XML`. +- `--json-results ` - write a JSON report (the same result that is printed + to stdout, wrapped in a summary). Also settable as `JSON_RESULTS`. + +See [Adding pluginval to CI](Adding%20pluginval%20to%20CI.md#machine-readable-reports) +for the report formats. The render runs on a background thread, as a host's audio thread would, leaving the message thread free for the plugin. diff --git a/docs/Adding pluginval to CI.md b/docs/Adding pluginval to CI.md index 3baa06b..43a63e4 100644 --- a/docs/Adding pluginval to CI.md +++ b/docs/Adding pluginval to CI.md @@ -41,3 +41,62 @@ $ ./pluginval validate --output-dir "./bin" "" || exit 1 ```sh > choco install pluginval ``` + +### Machine-readable reports +Both `pluginval validate` and `pluginval test` can write their results to a file, as well as printing them: + + - `--junit-xml ` writes a JUnit XML report, which most CI systems (GitHub Actions, GitLab, Jenkins, Azure Pipelines) can display as per-test results + - `--json-results ` writes the same results as JSON, for scripts and dashboards + +Relative paths are resolved against the working directory and any missing directories are created. Both options can be used together, and can also be set as the `JUNIT_XML` and `JSON_RESULTS` environment variables. The text output and log files are unchanged. If a report can't be written (e.g. the path isn't writable) the run fails with exit code `1`. + +```sh +$ ./pluginval validate --strictness-level 5 --junit-xml "./reports/MyPlugin.xml" --json-results "./reports/MyPlugin.json" "" +``` + +Each test pluginval runs becomes one ``, with a `` holding the failure messages if it failed: + +```xml + + + + + ... + + + +``` + +The JSON report has the same content: + +```json +{ + "pluginval_version": "2.0.0", + "command": "validate", + "target": "/path/to/MyPlugin.vst3", + "start_time": "...", + "duration_ms": 12345, + "properties": { "plugin_name": "MyPlugin", "strictness_level": "5" }, + "summary": { "tests": 20, "passed": 19, "failed": 1, "errors": 0 }, + "tests": [ + { "name": "Open plugin (cold)", "status": "passed", "duration_ms": 150, "messages": [] } + ] +} +``` + +`status` is `passed`, `failed` or `error`. For `pluginval test`, each test also has a `details` object holding the comparison results. + +If pluginval doesn't finish (the plugin crashes it, or the run times out), the report contains a single `error` entry saying so, so a report left over from an earlier run is never mistaken for a pass. + +##### GitHub Actions +```yaml +- name: Validate plugin + run: ./pluginval validate --strictness-level 5 --skip-gui-tests --junit-xml reports/pluginval.xml ./build/MyPlugin.vst3 + +- name: Upload pluginval report + if: always() + uses: actions/upload-artifact@v4 + with: + name: pluginval-report + path: reports/ +``` diff --git a/docs/Command line options.md b/docs/Command line options.md index 4d1bc56..71e2128 100644 --- a/docs/Command line options.md +++ b/docs/Command line options.md @@ -18,6 +18,8 @@ COMMANDS: pass instead. --timeout-ms Fail if the run takes longer (default 30000, -1 = never). + --junit-xml Write a JUnit XML report. + --json-results Write a JSON report. Any other option is an error. The flat flags --validate , --run-tests and --strictness-help [level] @@ -46,6 +48,8 @@ OPTIONS (for the validate command): --output-dir TEXT Directory in which to write the log files. --output-filename TEXT Filename to write logs into. + --junit-xml TEXT File to write a JUnit XML report to. + --json-results TEXT File to write a JSON report to. --disabled-tests TEXT Comma-separated test names, or a path to a file listing them. --random-seed TEXT Random seed (hex 0x.. or int) for replicable test runs. diff --git a/source/CommandLine.cpp b/source/CommandLine.cpp index f317557..9736bd1 100644 --- a/source/CommandLine.cpp +++ b/source/CommandLine.cpp @@ -232,6 +232,14 @@ void performCommandLine (CommandLineValidator& validator, const juce::String& co options.recordMissing = routed.recordMissing; options.timeoutMs = routed.timeoutMs; + const auto toReportFile = [] (const juce::String& path) + { + return path.isEmpty() ? juce::File() : juce::File::getCurrentWorkingDirectory().getChildFile (path); + }; + + options.junitXmlFile = toReportFile (routed.junitXmlPath); + options.jsonResultsFile = toReportFile (routed.jsonResultsPath); + // Runs async so will quit itself when done validator.runAcceptanceTest (juce::File::getCurrentWorkingDirectory().getChildFile (routed.testConfigPath), options); return; diff --git a/source/CommandLineTests.cpp b/source/CommandLineTests.cpp index 70b5d51..78ea4d3 100644 --- a/source/CommandLineTests.cpp +++ b/source/CommandLineTests.cpp @@ -61,6 +61,8 @@ struct CommandLineTests : public juce::UnitTest expect (opts.realtimeCheck == RealtimeCheck::disabled); expect (opts.dataFile == juce::File()); expect (opts.outputDir == juce::File()); + expect (opts.junitXmlFile == juce::File()); + expect (opts.jsonResultsFile == juce::File()); expect (opts.sampleRates == PluginvalSettings::defaultSampleRates()); expect (opts.blockSizes == PluginvalSettings::defaultBlockSizes()); } @@ -260,6 +262,39 @@ struct CommandLineTests : public juce::UnitTest } } + beginTest ("Report options (--junit-xml, --json-results)"); + { + const auto cwd = juce::File::getCurrentWorkingDirectory(); + + // Command line: relative paths resolve against the working directory. + { + const auto settings = parse ("--junit-xml out/report.xml --json-results out/report.json --validate x"); + expectEquals (juce::String (settings.junitXml), juce::String ("out/report.xml")); + expectEquals (juce::String (settings.jsonResults), juce::String ("out/report.json")); + + const auto opts = settings.toPluginTestOptions(); + expect (opts.junitXmlFile == cwd.getChildFile ("out/report.xml")); + expect (opts.jsonResultsFile == cwd.getChildFile ("out/report.json")); + } + + // Environment variables (names derived from the options). + { + const auto settings = parse ("--validate x", envFrom ({ { "JUNIT_XML", "env.xml" }, { "JSON_RESULTS", "env.json" } })); + expectEquals (juce::String (settings.junitXml), juce::String ("env.xml")); + expectEquals (juce::String (settings.jsonResults), juce::String ("env.json")); + } + + // --config keys mirror the field names. + { + juce::TemporaryFile configFile (".json"); + configFile.getFile().replaceWithText (R"({ "junitXml": "cfg.xml", "jsonResults": "cfg.json" })"); + + const auto settings = parse ("--config " + configFile.getFile().getFullPathName().quoted() + " --validate x"); + expectEquals (juce::String (settings.junitXml), juce::String ("cfg.xml")); + expectEquals (juce::String (settings.jsonResults), juce::String ("cfg.json")); + } + } + beginTest ("Repeatable --config merges per key, last wins"); { juce::TemporaryFile baseFile (".json"); @@ -288,6 +323,8 @@ struct CommandLineTests : public juce::UnitTest opts.randomiseTestOrder = true; opts.withGUI = false; opts.outputFilename = "log.txt"; + opts.junitXmlFile = juce::File::getCurrentWorkingDirectory().getChildFile ("report.xml"); + opts.jsonResultsFile = juce::File::getCurrentWorkingDirectory().getChildFile ("report.json"); opts.disabledTests = juce::StringArray ({ "Test A", "Test B" }); opts.sampleRates = { 44100.0, 96000.0 }; opts.blockSizes = { 64, 512 }; @@ -506,6 +543,186 @@ struct CommandLineTests : public juce::UnitTest } } + beginTest ("Subcommand: test report options (--junit-xml, --json-results)"); + { + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test --junit-xml out/r.xml config.json --json-results out/r.json")); + expect (d.error.isEmpty()); + expectEquals (d.junitXmlPath, juce::String ("out/r.xml")); + expectEquals (d.jsonResultsPath, juce::String ("out/r.json")); + expectEquals (d.testConfigPath, juce::String ("config.json")); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json --junit-xml")); + expect (d.error.contains ("--junit-xml")); + } + { + // The following option isn't taken as the file path. + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json --json-results --record-missing")); + expect (d.error.contains ("--json-results")); + } + } + + beginTest ("Report writer: JUnit XML"); + { + report_writer::Report report; + report.command = "validate"; + report.target = "MyPlugin"; + report.properties.push_back ({ "strictness_level", "5" }); + + report_writer::TestCase passed; + passed.name = "Open plugin (cold)"; + passed.durationSeconds = 0.25; + + report_writer::TestCase failed; + failed.name = "Plugin state"; + failed.status = report_writer::TestCase::Status::failed; + failed.messages.add ("Expected <1 & \"2\">"); + failed.messages.add ("second message"); + failed.durationSeconds = 1.0; + + report_writer::TestCase errored; + errored.name = "Editor"; + errored.status = report_writer::TestCase::Status::error; + errored.messages.add ("couldn't run"); + + report.cases = { passed, failed, errored }; + + const auto xml = juce::parseXML (report_writer::toJUnitXml (report)); + expect (xml != nullptr, "the report must be well-formed XML"); + + if (xml != nullptr) + { + expect (xml->hasTagName ("testsuites")); + expectEquals (xml->getIntAttribute ("tests"), 3); + expectEquals (xml->getIntAttribute ("failures"), 1); + expectEquals (xml->getIntAttribute ("errors"), 1); + expectEquals (xml->getStringAttribute ("time"), juce::String ("1.250")); + + const auto suite = xml->getChildByName ("testsuite"); + expect (suite != nullptr); + + if (suite != nullptr) + { + expectEquals (suite->getStringAttribute ("name"), juce::String ("pluginval validate: MyPlugin")); + expectEquals (suite->getIntAttribute ("tests"), 3); + expectEquals (suite->getNumChildElements(), 4); // properties + 3 testcases + + const auto property = suite->getChildByName ("properties")->getChildByName ("property"); + expectEquals (property->getStringAttribute ("name"), juce::String ("strictness_level")); + expectEquals (property->getStringAttribute ("value"), juce::String ("5")); + + const auto case0 = suite->getChildElement (1); + expectEquals (case0->getStringAttribute ("classname"), juce::String ("pluginval.validate")); + expectEquals (case0->getStringAttribute ("name"), juce::String ("Open plugin (cold)")); + expectEquals (case0->getStringAttribute ("time"), juce::String ("0.250")); + expectEquals (case0->getNumChildElements(), 0); + + // Special characters survive the round trip (i.e. were escaped). + const auto failure = suite->getChildElement (2)->getChildByName ("failure"); + expect (failure != nullptr); + + if (failure != nullptr) + { + expectEquals (failure->getStringAttribute ("message"), juce::String ("Expected <1 & \"2\">")); + expectEquals (failure->getAllSubText(), juce::String ("Expected <1 & \"2\">\nsecond message")); + } + + expect (suite->getChildElement (3)->getChildByName ("error") != nullptr); + expect (suite->getChildElement (3)->getChildByName ("failure") == nullptr); + } + } + } + + beginTest ("Report writer: JSON"); + { + report_writer::Report report; + report.command = "test"; + report.target = "sine-440"; + report.properties.push_back ({ "reference", "/refs/sine-440.wav" }); + + report_writer::TestCase passed; + passed.name = "a"; + passed.durationSeconds = 0.5; + + report_writer::TestCase failed; + failed.name = "b"; + failed.status = report_writer::TestCase::Status::failed; + failed.messages.add ("[sample] too different"); + failed.details = { { "outcome", "failed" } }; + + report.cases = { passed, failed }; + + const auto j = report_writer::toJson (report); + expectEquals (juce::String (j["command"].get()), juce::String ("test")); + expectEquals (juce::String (j["target"].get()), juce::String ("sine-440")); + expectEquals (juce::String (j["properties"]["reference"].get()), juce::String ("/refs/sine-440.wav")); + expectEquals (j["duration_ms"].get(), 500); + expectEquals (j["summary"]["tests"].get(), 2); + expectEquals (j["summary"]["passed"].get(), 1); + expectEquals (j["summary"]["failed"].get(), 1); + expectEquals (j["summary"]["errors"].get(), 0); + expectEquals ((int) j["tests"].size(), 2); + expectEquals (juce::String (j["tests"][0]["status"].get()), juce::String ("passed")); + expect (! j["tests"][0].contains ("details")); + expectEquals (juce::String (j["tests"][1]["status"].get()), juce::String ("failed")); + expectEquals (juce::String (j["tests"][1]["messages"][0].get()), juce::String ("[sample] too different")); + expectEquals (juce::String (j["tests"][1]["details"]["outcome"].get()), juce::String ("failed")); + } + + beginTest ("Report writer: incomplete report and file output"); + { + const auto report = report_writer::makeIncomplete ("validate", "MyPlugin", "did not complete"); + const auto j = report_writer::toJson (report); + expectEquals (j["summary"]["tests"].get(), 1); + expectEquals (j["summary"]["errors"].get(), 1); + expectEquals (juce::String (j["tests"][0]["messages"][0].get()), juce::String ("did not complete")); + + // Files are written (creating parent directories); unset files are skipped. + const auto dir = juce::File::getSpecialLocation (juce::File::tempDirectory) + .getChildFile ("pluginval_report_test_" + juce::String::toHexString (juce::Random::getSystemRandom().nextInt())); + const auto xmlFile = dir.getChildFile ("nested/report.xml"); + const auto jsonFile = dir.getChildFile ("nested/report.json"); + + expect (report_writer::write (report, {}, {})); + expect (! dir.exists()); + + expect (report_writer::write (report, xmlFile, jsonFile)); + expect (juce::parseXML (xmlFile) != nullptr); + expectEquals (nlohmann::json::parse (jsonFile.loadFileAsString().toStdString())["summary"]["errors"].get(), 1); + + dir.deleteRecursively(); + } + + beginTest ("Acceptance result to report"); + { + acceptance::TestResult result; + result.name = "sine-440"; + result.outcome = acceptance::TestResult::Outcome::failed; + result.durationSeconds = 2.0; + + acceptance::ComparisonResult comparison; + comparison.passed = false; + comparison.summary = "max abs diff 0.5"; + result.comparisons.push_back ({ "sample", comparison }); + + const auto failed = acceptance::reporter::toReport (result); + expectEquals (failed.command, juce::String ("test")); + expectEquals ((int) failed.cases.size(), 1); + expect (failed.cases[0].status == report_writer::TestCase::Status::failed); + expectEquals (failed.cases[0].messages[0], juce::String ("[sample] max abs diff 0.5")); + expectEquals (failed.cases[0].durationSeconds, 2.0); + expectEquals (juce::String (failed.cases[0].details["outcome"].get()), juce::String ("failed")); + + const auto error = acceptance::reporter::toReport (acceptance::TestResult::makeError ("t", "no plugin")); + expect (error.cases[0].status == report_writer::TestCase::Status::error); + expectEquals (error.cases[0].messages[0], juce::String ("no plugin")); + + result.outcome = acceptance::TestResult::Outcome::referenceCreated; + result.comparisons.clear(); + expect (acceptance::reporter::toReport (result).cases[0].status == report_writer::TestCase::Status::passed); + } + beginTest ("Acceptance TestConfig JSON parsing (snake_case keys, defaults)"); { const auto json = R"({ diff --git a/source/PluginTests.h b/source/PluginTests.h index 560893a..34b1910 100644 --- a/source/PluginTests.h +++ b/source/PluginTests.h @@ -47,6 +47,8 @@ struct PluginTests : public juce::UnitTest juce::File dataFile; /**< juce::File which tests can use to run user provided data. */ juce::File outputDir; /**< Directory in which to write the log files for each test run. */ juce::String outputFilename; /**< Filename to write logs into */ + juce::File junitXmlFile; /**< File to write a JUnit XML report to (none if default). */ + juce::File jsonResultsFile; /**< File to write a JSON report to (none if default). */ juce::StringArray disabledTests; /**< List of disabled tests. */ std::vector sampleRates; /**< List of sample rates. */ std::vector blockSizes; /**< List of block sizes. */ diff --git a/source/PluginvalSettings.h b/source/PluginvalSettings.h index e8a8255..e4afda6 100644 --- a/source/PluginvalSettings.h +++ b/source/PluginvalSettings.h @@ -54,6 +54,8 @@ struct PluginvalSettings std::string dataFile; std::string outputDir; std::string outputFilename; + std::string junitXml; + std::string jsonResults; std::vector disabledTests; std::vector sampleRates; std::vector blockSizes; @@ -83,6 +85,8 @@ struct PluginvalSettings o.dataFile = juce::String (dataFile); o.outputDir = juce::String (outputDir); o.outputFilename = juce::String (outputFilename); + o.junitXmlFile = toReportFile (junitXml); + o.jsonResultsFile = toReportFile (jsonResults); o.disabledTests = toStringArray (disabledTests); o.sampleRates = sampleRates.empty() ? defaultSampleRates() : sampleRates; o.blockSizes = blockSizes.empty() ? defaultBlockSizes() : blockSizes; @@ -106,6 +110,8 @@ struct PluginvalSettings s.dataFile = o.dataFile.getFullPathName().toStdString(); s.outputDir = o.outputDir.getFullPathName().toStdString(); s.outputFilename = o.outputFilename.toStdString(); + s.junitXml = o.junitXmlFile.getFullPathName().toStdString(); + s.jsonResults = o.jsonResultsFile.getFullPathName().toStdString(); s.disabledTests = fromStringArray (o.disabledTests); s.sampleRates = o.sampleRates; s.blockSizes = o.blockSizes; @@ -115,6 +121,15 @@ struct PluginvalSettings } private: + /** Resolves a report path against the working directory ("" means no report). */ + static juce::File toReportFile (const std::string& path) + { + if (path.empty()) + return {}; + + return juce::File::getCurrentWorkingDirectory().getChildFile (juce::String (path)); + } + static juce::StringArray toStringArray (const std::vector& v) { juce::StringArray a; @@ -139,4 +154,5 @@ struct PluginvalSettings NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT (PluginvalSettings, strictnessLevel, randomSeed, timeoutMs, verbose, numRepeats, randomiseTestOrder, skipGuiTests, dataFile, outputDir, outputFilename, - disabledTests, sampleRates, blockSizes, realtimeCheck, validatePath) + junitXml, jsonResults, disabledTests, sampleRates, blockSizes, realtimeCheck, + validatePath) diff --git a/source/ReportWriter.cpp b/source/ReportWriter.cpp new file mode 100644 index 0000000..7d70278 --- /dev/null +++ b/source/ReportWriter.cpp @@ -0,0 +1,217 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#include "ReportWriter.h" + +#include +#include + +namespace report_writer +{ + +//============================================================================== +namespace +{ + struct Totals + { + int tests = 0, failed = 0, errors = 0; + double seconds = 0.0; + }; + + Totals getTotals (const Report& report) + { + Totals t; + + for (const auto& c : report.cases) + { + ++t.tests; + t.failed += c.status == TestCase::Status::failed ? 1 : 0; + t.errors += c.status == TestCase::Status::error ? 1 : 0; + t.seconds += c.durationSeconds; + } + + return t; + } + + const char* toString (TestCase::Status s) + { + switch (s) + { + case TestCase::Status::passed: return "passed"; + case TestCase::Status::failed: return "failed"; + case TestCase::Status::error: return "error"; + } + + return "error"; + } + + juce::String formatSeconds (double seconds) + { + return juce::String (seconds, 3); + } + + juce::int64 toMilliseconds (double seconds) + { + return (juce::int64) std::llround (seconds * 1000.0); + } +} + +//============================================================================== +juce::String toJUnitXml (const Report& report) +{ + const auto totals = getTotals (report); + + const auto setTotals = [&totals] (juce::XmlElement& e) + { + e.setAttribute ("tests", totals.tests); + e.setAttribute ("failures", totals.failed); + e.setAttribute ("errors", totals.errors); + e.setAttribute ("time", formatSeconds (totals.seconds)); + }; + + // A plugin path makes for an unwieldy suite name, so use just the file name + const auto shortTarget = juce::File::isAbsolutePath (report.target) ? juce::File (report.target).getFileName() + : report.target; + + juce::XmlElement root ("testsuites"); + root.setAttribute ("name", "pluginval"); + setTotals (root); + + auto suite = root.createNewChildElement ("testsuite"); + suite->setAttribute ("name", "pluginval " + report.command + ": " + shortTarget); + setTotals (*suite); + suite->setAttribute ("skipped", 0); + suite->setAttribute ("timestamp", report.startTime.toISO8601 (true)); + + if (! report.properties.empty()) + { + auto properties = suite->createNewChildElement ("properties"); + + for (const auto& [name, value] : report.properties) + { + auto property = properties->createNewChildElement ("property"); + property->setAttribute ("name", name); + property->setAttribute ("value", value); + } + } + + for (const auto& c : report.cases) + { + auto testCase = suite->createNewChildElement ("testcase"); + + // A fixed classname: plugin names contain dots, which JUnit viewers + // treat as package separators + testCase->setAttribute ("classname", "pluginval." + report.command); + testCase->setAttribute ("name", c.name); + testCase->setAttribute ("time", formatSeconds (c.durationSeconds)); + + if (c.status != TestCase::Status::passed) + { + auto problem = testCase->createNewChildElement (c.status == TestCase::Status::failed ? "failure" : "error"); + problem->setAttribute ("message", c.messages[0].upToFirstOccurrenceOf ("\n", false, false)); + problem->addTextElement (c.messages.joinIntoString ("\n")); + } + } + + return root.toString(); +} + +nlohmann::ordered_json toJson (const Report& report) +{ + const auto totals = getTotals (report); + + nlohmann::ordered_json j; + j["pluginval_version"] = VERSION; + j["command"] = report.command.toStdString(); + j["target"] = report.target.toStdString(); + j["start_time"] = report.startTime.toISO8601 (true).toStdString(); + j["duration_ms"] = toMilliseconds (totals.seconds); + + auto properties = nlohmann::ordered_json::object(); + + for (const auto& [name, value] : report.properties) + properties[name.toStdString()] = value.toStdString(); + + j["properties"] = properties; + + j["summary"] = { { "tests", totals.tests }, + { "passed", totals.tests - totals.failed - totals.errors }, + { "failed", totals.failed }, + { "errors", totals.errors } }; + + auto tests = nlohmann::ordered_json::array(); + + for (const auto& c : report.cases) + { + nlohmann::ordered_json test; + test["name"] = c.name.toStdString(); + test["status"] = toString (c.status); + test["duration_ms"] = toMilliseconds (c.durationSeconds); + + auto messages = nlohmann::ordered_json::array(); + + for (const auto& m : c.messages) + messages.push_back (m.toStdString()); + + test["messages"] = messages; + + if (! c.details.is_null()) + test["details"] = c.details; + + tests.push_back (test); + } + + j["tests"] = tests; + return j; +} + +Report makeIncomplete (const juce::String& command, const juce::String& target, const juce::String& reason) +{ + TestCase c; + c.name = "pluginval " + command; + c.status = TestCase::Status::error; + c.messages.add (reason); + + Report report; + report.command = command; + report.target = target; + report.cases.push_back (c); + return report; +} + +//============================================================================== +bool write (const Report& report, const juce::File& junitXml, const juce::File& jsonResults) +{ + const auto writeFile = [] (const juce::File& file, const juce::String& text) + { + if (file.getParentDirectory().createDirectory().wasOk() + && file.replaceWithText (text, false, false, "\n")) + return true; + + std::cerr << "!!! WARNING: Unable to write the report to " << file.getFullPathName() << std::endl; + return false; + }; + + bool ok = true; + + if (junitXml != juce::File()) + ok = writeFile (junitXml, toJUnitXml (report)) && ok; + + if (jsonResults != juce::File()) + ok = writeFile (jsonResults, juce::String::fromUTF8 (toJson (report).dump (2, ' ', false, nlohmann::ordered_json::error_handler_t::replace).c_str()) + "\n") && ok; + + return ok; +} + +} // namespace report_writer diff --git a/source/ReportWriter.h b/source/ReportWriter.h new file mode 100644 index 0000000..a5e7104 --- /dev/null +++ b/source/ReportWriter.h @@ -0,0 +1,78 @@ +/*============================================================================== + + Copyright 2018 by Tracktion Corporation. + For more information visit www.tracktion.com + + You may also use this code under the terms of the GPL v3 (see + www.gnu.org/licenses). + + pluginval IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER + EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE + DISCLAIMED. + + ==============================================================================*/ + +#pragma once + +#include +#include + +#include +#include + +//============================================================================== +/** + Machine-readable reports (JUnit XML and JSON) shared by the validate and + test commands. Each command converts its own results into a Report, which is + then written to whichever files were requested (--junit-xml / --json-results). +*/ +namespace report_writer +{ + //============================================================================== + /** One test within a run. */ + struct TestCase + { + enum class Status + { + passed, + failed, /**< The test ran and a check failed. */ + error /**< The test couldn't be run to completion. */ + }; + + juce::String name; + Status status = Status::passed; + juce::StringArray messages; /**< Failure / error text. */ + double durationSeconds = 0.0; + nlohmann::json details; /**< Optional command-specific data (JSON report only). */ + }; + + /** The results of one run of a command. */ + struct Report + { + juce::String command; /**< "validate" or "test". */ + juce::String target; /**< Plugin path/ID, or the acceptance test name. */ + juce::Time startTime = juce::Time::getCurrentTime(); + std::vector> properties; + std::vector cases; + }; + + //============================================================================== + /** Returns the report as a JUnit XML document (a single testsuite). */ + juce::String toJUnitXml (const Report&); + + /** Returns the report as JSON (snake_case keys). */ + nlohmann::ordered_json toJson (const Report&); + + /** A report with a single error case, for a run that didn't finish. + Written before a run starts so a crash or timeout can't leave a stale + (or missing) report behind; a completed run overwrites it. + */ + Report makeIncomplete (const juce::String& command, const juce::String& target, const juce::String& reason); + + /** Writes the report to whichever files are set (a default juce::File is + skipped), creating parent directories. Returns false, with a warning on + stderr, if a file couldn't be written (which the commands treat as a + failed run). + */ + bool write (const Report&, const juce::File& junitXml, const juce::File& jsonResults); +} diff --git a/source/SettingsParser.cpp b/source/SettingsParser.cpp index 45b6135..999a6a6 100644 --- a/source/SettingsParser.cpp +++ b/source/SettingsParser.cpp @@ -135,6 +135,8 @@ R"(Commands: (default: a missing reference fails). --timeout-ms Fail if the run takes longer than this (default 30000, -1 to never time out). + --junit-xml Write a JUnit XML report. + --json-results Write a JSON report. The flat flags --validate , --run-tests and --strictness-help [level] are deprecated aliases for the commands above and will be removed in a future version. @@ -195,6 +197,8 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma app.add_option ("--data-file", s.dataFile, "Path to a data file tests can use to configure themselves."); app.add_option ("--output-dir", s.outputDir, "Directory in which to write the log files."); app.add_option ("--output-filename", s.outputFilename, "Filename to write logs into."); + app.add_option ("--junit-xml", s.junitXml, "File to write a JUnit XML report to."); + app.add_option ("--json-results", s.jsonResults, "File to write a JSON report to."); app.add_option_function ("--disabled-tests", [&s] (const std::string& v) { s.disabledTests = settings_serializer::disabledTestsToList (juce::String (v)); }, @@ -382,6 +386,9 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma if (const auto env = juce::SystemStats::getEnvironmentVariable ("TIMEOUT_MS", {}); env.isNotEmpty()) result.timeoutMs = env.getLargeIntValue(); + result.junitXmlPath = juce::SystemStats::getEnvironmentVariable ("JUNIT_XML", {}); + result.jsonResultsPath = juce::SystemStats::getEnvironmentVariable ("JSON_RESULTS", {}); + // The options, plus the positional acceptance-test config (the first // non-option token after the verb). for (int i = 1; i < tokensIn.size(); ++i) @@ -405,6 +412,18 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma result.timeoutMs = value.getLargeIntValue(); } + else if (token == "--junit-xml" || token == "--json-results") + { + const auto value = i + 1 < tokensIn.size() ? tokensIn.getReference (++i) : juce::String(); + + if (value.isEmpty() || value.startsWith ("--")) + { + result.error = token + " needs a file path"; + break; + } + + (token == "--junit-xml" ? result.junitXmlPath : result.jsonResultsPath) = value; + } else if (token.startsWith ("-")) { result.error = "unknown option for test: " + token; diff --git a/source/SettingsParser.h b/source/SettingsParser.h index 5d5ae2a..934b0c0 100644 --- a/source/SettingsParser.h +++ b/source/SettingsParser.h @@ -83,6 +83,8 @@ namespace settings_parser juce::String testConfigPath; /**< The positional config.json for the test command. */ bool recordMissing = false; /**< test: record a missing reference (and pass) instead of failing. */ juce::int64 timeoutMs = 30000; /**< test: abort the run after this long (-1 to never time out). */ + juce::String junitXmlPath; /**< test: file to write a JUnit XML report to. */ + juce::String jsonResultsPath; /**< test: file to write a JSON report to. */ juce::String error; /**< test: an invalid option / argument, to report instead of running. */ }; diff --git a/source/Validator.cpp b/source/Validator.cpp index e9a6fdf..724b36a 100644 --- a/source/Validator.cpp +++ b/source/Validator.cpp @@ -16,6 +16,7 @@ #include "PluginTests.h" #include "CrashHandler.h" #include "CommandLine.h" +#include "ReportWriter.h" #include #include #include @@ -24,9 +25,11 @@ struct PluginsUnitTestRunner : public juce::UnitTestRunner, private juce::Thread { - PluginsUnitTestRunner (std::function logCallback, std::unique_ptr logDestination, juce::int64 timeoutInMs) + PluginsUnitTestRunner (std::function logCallback, std::unique_ptr logDestination, juce::int64 timeoutInMs, + std::function timeoutCallback = {}) : Thread ("TimoutThread"), callback (std::move (logCallback)), + onTimeout (std::move (timeoutCallback)), outputStream (std::move (logDestination)), timeoutMs (timeoutInMs) { @@ -73,6 +76,7 @@ struct PluginsUnitTestRunner : public juce::UnitTestRunner, private: std::function callback; + std::function onTimeout; // Called on the timeout thread just before the process is terminated std::unique_ptr outputStream; const juce::int64 timeoutMs = -1; std::atomic timoutTime { -1 }; @@ -95,6 +99,10 @@ struct PluginsUnitTestRunner : public juce::UnitTestRunner, // Give the log a second to flush the message before terminating juce::Thread::sleep (1000); + + if (onTimeout) + onTimeout(); + juce::Process::terminate(); } @@ -186,6 +194,47 @@ static void updateFileNameIfPossible (PluginTests& test, PluginsUnitTestRunner& } +//============================================================================== +/** Converts the results of a validation into a report for --junit-xml / --json-results. */ +static report_writer::Report toReport (PluginTests& test, const juce::Array& results) +{ + const auto options = test.getOptions(); + + report_writer::Report report; + report.command = "validate"; + report.target = test.getFileOrID(); + + if (! results.isEmpty()) + report.startTime = results.getReference (0).startTime; + + if (auto pd = test.getDescriptions().getFirst()) + { + report.properties.push_back ({ "plugin_name", pd->name }); + report.properties.push_back ({ "plugin_version", pd->version }); + report.properties.push_back ({ "plugin_format", pd->pluginFormatName }); + report.properties.push_back ({ "plugin_manufacturer", pd->manufacturerName }); + } + + report.properties.push_back ({ "strictness_level", juce::String (options.strictnessLevel) }); + report.properties.push_back ({ "timeout_ms", juce::String (options.timeoutMs) }); + + if (options.randomSeed != 0) + report.properties.push_back ({ "random_seed", "0x" + juce::String::toHexString (options.randomSeed) }); + + for (const auto& r : results) + { + report_writer::TestCase c; + c.name = r.subcategoryName; + c.status = r.failures > 0 ? report_writer::TestCase::Status::failed + : report_writer::TestCase::Status::passed; + c.messages = r.messages; + c.durationSeconds = juce::jmax (0.0, (r.endTime - r.startTime).inSeconds()); + report.cases.push_back (c); + } + + return report; +} + //============================================================================== //============================================================================== inline int getNumFailures (juce::Array results); @@ -194,9 +243,24 @@ inline juce::Array runTests (PluginTests& test { const auto options = test.getOptions(); juce::Array results; - PluginsUnitTestRunner testRunner (std::move (callback), createDestinationFileStream (test), options.timeoutMs); + const auto writeIncompleteReport = [options, fileOrID = test.getFileOrID()] (const juce::String& reason) + { + report_writer::write (report_writer::makeIncomplete ("validate", fileOrID, reason), + options.junitXmlFile, options.jsonResultsFile); + }; + + // Written last on a timeout, so the report always agrees with the exit code + // (the tests can finish in the moment between the timeout and the process ending) + PluginsUnitTestRunner testRunner (std::move (callback), createDestinationFileStream (test), options.timeoutMs, + [options, writeIncompleteReport] + { + writeIncompleteReport ("Validation timed out after " + juce::RelativeTime::milliseconds (options.timeoutMs).getDescription()); + }); testRunner.setAssertOnFailure (false); + // Written up front so a crash can't leave a stale (or missing) report behind + writeIncompleteReport ("Validation did not complete (crashed)"); + juce::Array testsToRun; testsToRun.add (&test); testRunner.runTests (testsToRun, options.randomSeed); @@ -205,6 +269,19 @@ inline juce::Array runTests (PluginTests& test results.add (*testRunner.getResult (i)); updateFileNameIfPossible (test, testRunner); + + // A report that was asked for but couldn't be written is a failure, so CI doesn't silently lose it + if (! report_writer::write (toReport (test, results), options.junitXmlFile, options.jsonResultsFile)) + { + juce::UnitTestRunner::TestResult writeFailure ("pluginval", "Write report files"); + writeFailure.failures = 1; + writeFailure.messages.add ("Unable to write the report file(s)"); + writeFailure.endTime = writeFailure.startTime; + results.add (writeFailure); + + testRunner.logMessage ("*** FAILED: Unable to write the report file(s)"); + } + const int failures = getNumFailures (results); if (! failures) diff --git a/source/acceptance/AcceptanceTest.cpp b/source/acceptance/AcceptanceTest.cpp index 26d67a8..100cdfb 100644 --- a/source/acceptance/AcceptanceTest.cpp +++ b/source/acceptance/AcceptanceTest.cpp @@ -593,6 +593,23 @@ TestResult runTest (const TestConfig& config, const RunOptions& options) //============================================================================== int runTestFile (const juce::File& configFile, const RunOptions& options) { + const auto finish = [&options] (const TestResult& result) + { + // A report that was asked for but couldn't be written is a failure + if (! reporter::report (result, options.junitXmlFile, options.jsonResultsFile)) + { + std::cout << "*** FAILED: Unable to write the report file(s)" << std::endl; + return 1; + } + + return reporter::exitCode (result); + }; + + // Written up front so a crash can't leave a stale (or missing) report behind + report_writer::write (report_writer::makeIncomplete ("test", configFile.getFileNameWithoutExtension(), + "Test did not complete (crashed)"), + options.junitXmlFile, options.jsonResultsFile); + std::vector configs; try @@ -601,33 +618,28 @@ int runTestFile (const juce::File& configFile, const RunOptions& options) } catch (const std::exception& e) { - const auto result = TestResult::makeError (configFile.getFileNameWithoutExtension(), e.what()); - reporter::report (result); - return reporter::exitCode (result); + return finish (TestResult::makeError (configFile.getFileNameWithoutExtension(), e.what())); } if (configs.empty()) - { - const auto result = TestResult::makeError (configFile.getFileNameWithoutExtension(), "no test configs in file"); - reporter::report (result); - return reporter::exitCode (result); - } + return finish (TestResult::makeError (configFile.getFileNameWithoutExtension(), "no test configs in file")); // v1 runs the first entry only; the parser already accepts arrays for the // phase-2 multiplexing extension. if (configs.size() > 1) std::cout << "Note: " << configs.size() << " configs found; v1 runs the first only." << std::endl; - const auto result = runTest (configs.front(), options); - reporter::report (result); - return reporter::exitCode (result); + const auto startTime = juce::Time::getMillisecondCounterHiRes(); + auto result = runTest (configs.front(), options); + result.durationSeconds = (juce::Time::getMillisecondCounterHiRes() - startTime) / 1000.0; + return finish (result); } //============================================================================== struct TestRunner::Watchdog : private juce::Thread { - Watchdog (const juce::File& file, juce::int64 timeout) - : Thread ("pluginval test watchdog"), configFile (file), timeoutMs (timeout) + Watchdog (const juce::File& file, const RunOptions& opts) + : Thread ("pluginval test watchdog"), configFile (file), options (opts) { startThread (juce::Thread::Priority::low); } @@ -645,19 +657,20 @@ struct TestRunner::Watchdog : private juce::Thread private: const juce::File configFile; - const juce::int64 timeoutMs; + const RunOptions options; juce::WaitableEvent finished; void run() override { - if (finished.wait ((double) timeoutMs)) + if (finished.wait ((double) options.timeoutMs)) return; // The worker is stuck in a plugin call that can't be interrupted, so // report the failure and end the process. - const auto result = TestResult::makeError (configFile.getFileNameWithoutExtension(), - "timed out after " + juce::RelativeTime::milliseconds (timeoutMs).getDescription()); - reporter::report (result); + auto result = TestResult::makeError (configFile.getFileNameWithoutExtension(), + "timed out after " + juce::RelativeTime::milliseconds (options.timeoutMs).getDescription()); + result.durationSeconds = (double) options.timeoutMs / 1000.0; + reporter::report (result, options.junitXmlFile, options.jsonResultsFile); std::cout << std::flush; std::_Exit (reporter::exitCode (result)); } @@ -673,7 +686,7 @@ TestRunner::TestRunner (const juce::File& file, const RunOptions& opts, std::fun jassert (onComplete); if (options.timeoutMs > 0) - watchdog = std::make_unique (configFile, options.timeoutMs); + watchdog = std::make_unique (configFile, options); startThread(); } diff --git a/source/acceptance/AcceptanceTest.h b/source/acceptance/AcceptanceTest.h index 65ea6be..a8d1d93 100644 --- a/source/acceptance/AcceptanceTest.h +++ b/source/acceptance/AcceptanceTest.h @@ -52,6 +52,8 @@ struct RunOptions { bool recordMissing = false; /**< Record a missing reference and pass, rather than failing. */ juce::int64 timeoutMs = 30000; /**< Abort the whole run after this long. <= 0 never times out. */ + juce::File junitXmlFile; /**< File to write a JUnit XML report to (none if default). */ + juce::File jsonResultsFile; /**< File to write a JSON report to (none if default). */ }; //============================================================================== @@ -60,7 +62,8 @@ struct RunOptions TestResult runTest (const TestConfig&, const RunOptions&); /** Loads the config file (first entry only for v1), runs it, reports the result - to stdout and returns the process exit code (0 success, 1 failure). */ + to stdout (and any report files in the options) and returns the process exit + code (0 success, 1 failure). */ int runTestFile (const juce::File& configFile, const RunOptions&); //============================================================================== diff --git a/source/acceptance/TestReporter.cpp b/source/acceptance/TestReporter.cpp index a95c100..d495cf7 100644 --- a/source/acceptance/TestReporter.cpp +++ b/source/acceptance/TestReporter.cpp @@ -70,7 +70,43 @@ nlohmann::json toJson (const TestResult& r) return j; } -void report (const TestResult& r) +report_writer::Report toReport (const TestResult& r) +{ + report_writer::TestCase c; + c.name = r.name; + c.durationSeconds = r.durationSeconds; + c.details = toJson (r); + + if (r.outcome == TestResult::Outcome::failed) + { + c.status = report_writer::TestCase::Status::failed; + + for (const auto& [method, result] : r.comparisons) + if (! result.passed) + c.messages.add ("[" + method + "] " + result.summary); + + if (r.diffFile != juce::File()) + c.messages.add ("diff written to: " + r.diffFile.getFullPathName()); + } + else if (r.outcome == TestResult::Outcome::error) + { + c.status = report_writer::TestCase::Status::error; + c.messages.add (r.message); + } + + report_writer::Report report; + report.command = "test"; + report.target = r.name; + report.startTime = juce::Time::getCurrentTime() - juce::RelativeTime::seconds (r.durationSeconds); + + if (r.referenceFile != juce::File()) + report.properties.push_back ({ "reference", r.referenceFile.getFullPathName() }); + + report.cases.push_back (c); + return report; +} + +bool report (const TestResult& r, const juce::File& junitXml, const juce::File& jsonResults) { switch (r.outcome) { @@ -99,6 +135,8 @@ void report (const TestResult& r) // The structured result for machine consumers. std::cout << toJson (r).dump (2) << std::endl; + + return report_writer::write (toReport (r), junitXml, jsonResults); } int exitCode (const TestResult& r) diff --git a/source/acceptance/TestReporter.h b/source/acceptance/TestReporter.h index 10266b7..4939fc3 100644 --- a/source/acceptance/TestReporter.h +++ b/source/acceptance/TestReporter.h @@ -15,6 +15,7 @@ #pragma once #include "ReferenceComparator.h" +#include "../ReportWriter.h" #include #include @@ -46,6 +47,8 @@ struct TestResult juce::File referenceFile; juce::File diffFile; /**< Written only on failure. */ + double durationSeconds = 0.0; /**< How long the test took to run. */ + //============================================================================== static TestResult makeError (const juce::String& name, const juce::String& message) { @@ -63,8 +66,13 @@ namespace reporter /** Structured, machine-readable representation of a result. */ nlohmann::json toJson (const TestResult&); - /** Prints a human-readable summary followed by the JSON to stdout. */ - void report (const TestResult&); + /** The result as a report for --junit-xml / --json-results. */ + report_writer::Report toReport (const TestResult&); + + /** Prints a human-readable summary followed by the JSON to stdout, then + writes the report files that are set (a default juce::File is skipped). + Returns false if a report file couldn't be written. */ + bool report (const TestResult&, const juce::File& junitXml = {}, const juce::File& jsonResults = {}); /** 0 for referenceCreated / passed, 1 for failed / error. */ int exitCode (const TestResult&); diff --git a/tests/acceptance/CMakeLists.txt b/tests/acceptance/CMakeLists.txt index 5d1f350..67e01b9 100644 --- a/tests/acceptance/CMakeLists.txt +++ b/tests/acceptance/CMakeLists.txt @@ -34,6 +34,10 @@ foreach(case ${acceptance_cases}) file(GENERATE OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/$/${case}.json" INPUT "${CMAKE_CURRENT_BINARY_DIR}/${case}.json.gen") + # The report options are passed so the JUnit / JSON writers are exercised too. add_test(NAME "pluginval.acceptance.${case}" - COMMAND pluginval test "${CMAKE_CURRENT_BINARY_DIR}/$/${case}.json") + COMMAND pluginval test + --junit-xml "${CMAKE_CURRENT_BINARY_DIR}/$/reports/${case}.junit.xml" + --json-results "${CMAKE_CURRENT_BINARY_DIR}/$/reports/${case}.json" + "${CMAKE_CURRENT_BINARY_DIR}/$/${case}.json") endforeach() From 2fb3c2456f9317fc5285675fea0dd5f50c22f020 Mon Sep 17 00:00:00 2001 From: David Rowland Date: Tue, 13 Oct 2026 13:48:05 +0100 Subject: [PATCH 2/3] Report output: Renamed TestCase::Status to avoid the X11 Status macro on Linux --- source/CommandLineTests.cpp | 12 ++++++------ source/ReportWriter.cpp | 18 +++++++++--------- source/ReportWriter.h | 4 ++-- source/Validator.cpp | 4 ++-- source/acceptance/TestReporter.cpp | 4 ++-- 5 files changed, 21 insertions(+), 21 deletions(-) diff --git a/source/CommandLineTests.cpp b/source/CommandLineTests.cpp index 78ea4d3..bb5fed9 100644 --- a/source/CommandLineTests.cpp +++ b/source/CommandLineTests.cpp @@ -576,14 +576,14 @@ struct CommandLineTests : public juce::UnitTest report_writer::TestCase failed; failed.name = "Plugin state"; - failed.status = report_writer::TestCase::Status::failed; + failed.status = report_writer::TestCase::Outcome::failed; failed.messages.add ("Expected <1 & \"2\">"); failed.messages.add ("second message"); failed.durationSeconds = 1.0; report_writer::TestCase errored; errored.name = "Editor"; - errored.status = report_writer::TestCase::Status::error; + errored.status = report_writer::TestCase::Outcome::error; errored.messages.add ("couldn't run"); report.cases = { passed, failed, errored }; @@ -647,7 +647,7 @@ struct CommandLineTests : public juce::UnitTest report_writer::TestCase failed; failed.name = "b"; - failed.status = report_writer::TestCase::Status::failed; + failed.status = report_writer::TestCase::Outcome::failed; failed.messages.add ("[sample] too different"); failed.details = { { "outcome", "failed" } }; @@ -709,18 +709,18 @@ struct CommandLineTests : public juce::UnitTest const auto failed = acceptance::reporter::toReport (result); expectEquals (failed.command, juce::String ("test")); expectEquals ((int) failed.cases.size(), 1); - expect (failed.cases[0].status == report_writer::TestCase::Status::failed); + expect (failed.cases[0].status == report_writer::TestCase::Outcome::failed); expectEquals (failed.cases[0].messages[0], juce::String ("[sample] max abs diff 0.5")); expectEquals (failed.cases[0].durationSeconds, 2.0); expectEquals (juce::String (failed.cases[0].details["outcome"].get()), juce::String ("failed")); const auto error = acceptance::reporter::toReport (acceptance::TestResult::makeError ("t", "no plugin")); - expect (error.cases[0].status == report_writer::TestCase::Status::error); + expect (error.cases[0].status == report_writer::TestCase::Outcome::error); expectEquals (error.cases[0].messages[0], juce::String ("no plugin")); result.outcome = acceptance::TestResult::Outcome::referenceCreated; result.comparisons.clear(); - expect (acceptance::reporter::toReport (result).cases[0].status == report_writer::TestCase::Status::passed); + expect (acceptance::reporter::toReport (result).cases[0].status == report_writer::TestCase::Outcome::passed); } beginTest ("Acceptance TestConfig JSON parsing (snake_case keys, defaults)"); diff --git a/source/ReportWriter.cpp b/source/ReportWriter.cpp index 7d70278..7d34240 100644 --- a/source/ReportWriter.cpp +++ b/source/ReportWriter.cpp @@ -36,21 +36,21 @@ namespace for (const auto& c : report.cases) { ++t.tests; - t.failed += c.status == TestCase::Status::failed ? 1 : 0; - t.errors += c.status == TestCase::Status::error ? 1 : 0; + t.failed += c.status == TestCase::Outcome::failed ? 1 : 0; + t.errors += c.status == TestCase::Outcome::error ? 1 : 0; t.seconds += c.durationSeconds; } return t; } - const char* toString (TestCase::Status s) + const char* toString (TestCase::Outcome s) { switch (s) { - case TestCase::Status::passed: return "passed"; - case TestCase::Status::failed: return "failed"; - case TestCase::Status::error: return "error"; + case TestCase::Outcome::passed: return "passed"; + case TestCase::Outcome::failed: return "failed"; + case TestCase::Outcome::error: return "error"; } return "error"; @@ -116,9 +116,9 @@ juce::String toJUnitXml (const Report& report) testCase->setAttribute ("name", c.name); testCase->setAttribute ("time", formatSeconds (c.durationSeconds)); - if (c.status != TestCase::Status::passed) + if (c.status != TestCase::Outcome::passed) { - auto problem = testCase->createNewChildElement (c.status == TestCase::Status::failed ? "failure" : "error"); + auto problem = testCase->createNewChildElement (c.status == TestCase::Outcome::failed ? "failure" : "error"); problem->setAttribute ("message", c.messages[0].upToFirstOccurrenceOf ("\n", false, false)); problem->addTextElement (c.messages.joinIntoString ("\n")); } @@ -180,7 +180,7 @@ Report makeIncomplete (const juce::String& command, const juce::String& target, { TestCase c; c.name = "pluginval " + command; - c.status = TestCase::Status::error; + c.status = TestCase::Outcome::error; c.messages.add (reason); Report report; diff --git a/source/ReportWriter.h b/source/ReportWriter.h index a5e7104..86268d7 100644 --- a/source/ReportWriter.h +++ b/source/ReportWriter.h @@ -32,7 +32,7 @@ namespace report_writer /** One test within a run. */ struct TestCase { - enum class Status + enum class Outcome { passed, failed, /**< The test ran and a check failed. */ @@ -40,7 +40,7 @@ namespace report_writer }; juce::String name; - Status status = Status::passed; + Outcome status = Outcome::passed; juce::StringArray messages; /**< Failure / error text. */ double durationSeconds = 0.0; nlohmann::json details; /**< Optional command-specific data (JSON report only). */ diff --git a/source/Validator.cpp b/source/Validator.cpp index 724b36a..6e4b262 100644 --- a/source/Validator.cpp +++ b/source/Validator.cpp @@ -225,8 +225,8 @@ static report_writer::Report toReport (PluginTests& test, const juce::Array 0 ? report_writer::TestCase::Status::failed - : report_writer::TestCase::Status::passed; + c.status = r.failures > 0 ? report_writer::TestCase::Outcome::failed + : report_writer::TestCase::Outcome::passed; c.messages = r.messages; c.durationSeconds = juce::jmax (0.0, (r.endTime - r.startTime).inSeconds()); report.cases.push_back (c); diff --git a/source/acceptance/TestReporter.cpp b/source/acceptance/TestReporter.cpp index d495cf7..4f48a34 100644 --- a/source/acceptance/TestReporter.cpp +++ b/source/acceptance/TestReporter.cpp @@ -79,7 +79,7 @@ report_writer::Report toReport (const TestResult& r) if (r.outcome == TestResult::Outcome::failed) { - c.status = report_writer::TestCase::Status::failed; + c.status = report_writer::TestCase::Outcome::failed; for (const auto& [method, result] : r.comparisons) if (! result.passed) @@ -90,7 +90,7 @@ report_writer::Report toReport (const TestResult& r) } else if (r.outcome == TestResult::Outcome::error) { - c.status = report_writer::TestCase::Status::error; + c.status = report_writer::TestCase::Outcome::error; c.messages.add (r.message); } From 7c83ee7814cd8b18ff6cfbe641010dbaafd26a4f Mon Sep 17 00:00:00 2001 From: David Rowland Date: Tue, 13 Oct 2026 13:56:44 +0100 Subject: [PATCH 3/3] Add --plugin to pluginval test and GitHub Actions for validate and acceptance tests --- .github/actions/README.md | 92 +++++++++++++++++ .github/actions/acceptance/acceptance.sh | 93 +++++++++++++++++ .github/actions/acceptance/action.yml | 51 ++++++++++ .github/actions/common/common.sh | 74 ++++++++++++++ .github/actions/validate/action.yml | 48 +++++++++ .github/actions/validate/validate.sh | 67 +++++++++++++ .github/workflows/acceptance.yml | 99 +++++++++++++++++++ .github/workflows/build.yaml | 2 + CHANGELIST.md | 2 + CLAUDE.md | 33 ++++++- docs/Acceptance testing.md | 17 +++- docs/Adding pluginval to CI.md | 15 +++ docs/Command line options.md | 2 + source/CommandLine.cpp | 1 + source/CommandLineTests.cpp | 25 ++++- source/SettingsParser.cpp | 14 +++ source/SettingsParser.h | 1 + source/acceptance/AcceptanceTest.cpp | 10 +- source/acceptance/AcceptanceTest.h | 1 + source/acceptance/TestConfig.cpp | 6 +- source/acceptance/TestConfig.h | 2 +- tests/acceptance/Acceptance testing design.md | 2 +- 22 files changed, 646 insertions(+), 11 deletions(-) create mode 100644 .github/actions/README.md create mode 100644 .github/actions/acceptance/acceptance.sh create mode 100644 .github/actions/acceptance/action.yml create mode 100644 .github/actions/common/common.sh create mode 100644 .github/actions/validate/action.yml create mode 100644 .github/actions/validate/validate.sh diff --git a/.github/actions/README.md b/.github/actions/README.md new file mode 100644 index 0000000..45264a9 --- /dev/null +++ b/.github/actions/README.md @@ -0,0 +1,92 @@ +# pluginval GitHub Actions + +Two composite actions for using pluginval from another repository's workflows. +Both run on Linux, macOS and Windows runners, download the matching pluginval +release (or use a binary you give them), run everything they are given, write a +results table to the job summary and fail the step if anything failed. + +On Linux the plugin is run under `xvfb-run` when there is no `DISPLAY` (install +`xvfb` on runners that don't have it). + +## validate + +Validates one or more plugins. + +```yaml +- uses: Tracktion/pluginval/.github/actions/validate@v2 + with: + plugins: | + build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3 + build/MyPlugin_artefacts/Release/AU/MyPlugin.component + strictness-level: 10 + args: --skip-gui-tests +``` + +| Input | Default | | +| --- | --- | --- | +| `plugins` | (required) | The plugins to validate, one per line. Each is a path, a glob or an AU id. A glob that matches nothing fails. | +| `strictness-level` | `5` | The strictness level (1-10). | +| `args` | | Extra options for `pluginval validate`, split on whitespace. | +| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. | +| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. | + +Every `pluginval validate` option can also be set as an environment variable on +the step (e.g. `env: { TIMEOUT_MS: 60000 }`), which is the way to pass values +containing spaces. + +## acceptance + +Runs the [acceptance tests](../../docs/Acceptance%20testing.md) for every config +(`*.json`) found recursively under a directory, optionally against one built +plugin. + +```yaml +- uses: Tracktion/pluginval/.github/actions/acceptance@v2 + with: + config-dir: tests/acceptance + plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3 +``` + +| Input | Default | | +| --- | --- | --- | +| `config-dir` | (required) | Directory searched recursively for configs. | +| `plugin` | | Plugin path or AU id to test every config against, instead of the `plugin` in each config. | +| `record-missing` | `false` | Record a missing reference and pass, instead of failing. | +| `timeout-ms` | | Fail a test if it takes longer than this (pluginval's default if empty). | +| `pluginval-version` | `v2.0.0-beta1` | The pluginval release to download: a tag, or `latest`. | +| `pluginval-path` | | An existing pluginval binary to use instead of downloading one. | + +Notes: + +- Every `*.json` under `config-dir` is treated as a config, apart from the + `.wav.json` manifests. Keep other JSON files (and plugin bundles, + which contain their own) out of it. +- A config holding an array of tests is reported as a failure: use one config + per file. +- Relative `plugin` paths resolve against the working directory; the input, + state and reference paths in a config resolve against the config's directory. + +## Outputs + +Both actions set: + +| Output | | +| --- | --- | +| `passed` / `failed` | How many plugins / configs passed and failed. | +| `report-dir` | A directory holding a JUnit XML report for each one. For `acceptance` it also has a `diffs` directory with the `*-diff.wav` files found under `config-dir`. | + +To keep the reports and diffs from a failed run: + +```yaml +- uses: Tracktion/pluginval/.github/actions/acceptance@v2 + id: acceptance + with: + config-dir: tests/acceptance + plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3 + +- uses: actions/upload-artifact@v4 + if: always() + with: + name: pluginval-acceptance-${{ runner.os }} + path: ${{ steps.acceptance.outputs.report-dir }} +``` diff --git a/.github/actions/acceptance/acceptance.sh b/.github/actions/acceptance/acceptance.sh new file mode 100644 index 0000000..68dd63d --- /dev/null +++ b/.github/actions/acceptance/acceptance.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# Runs "pluginval test" on every config found (recursively) under +# INPUT_CONFIG_DIR, runs them all, and fails if any failed. See action.yml for +# the inputs. + +set -u +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../common/common.sh" + +resolve_pluginval + +config_dir="${INPUT_CONFIG_DIR:-}" + +if [ ! -d "$config_dir" ]; then + echo "::error::config-dir is not a directory: $config_dir" + exit 1 +fi + +report_dir="${INPUT_REPORT_DIR:-$RUNNER_TEMP/pluginval-reports/acceptance}" +mkdir -p "$report_dir" + +options=() +[ -n "${INPUT_PLUGIN:-}" ] && options+=(--plugin "$INPUT_PLUGIN") +[ "${INPUT_RECORD_MISSING:-false}" = "true" ] && options+=(--record-missing) +[ -n "${INPUT_TIMEOUT_MS:-}" ] && options+=(--timeout-ms "$INPUT_TIMEOUT_MS") + +passed=0 +failed=0 +index=0 +rows="" + +# Every *.json is a config, except the .wav.json manifests that +# pluginval writes next to each reference. +IFS=$'\n' +configs=( $(find "$config_dir" -type f -name '*.json' ! -name '*.wav.json' | sort) ) +unset IFS + +for config in ${configs[@]+"${configs[@]}"}; do + index=$((index + 1)) + name="${config#"$config_dir"/}" + report="$report_dir/$index-$(sanitise "$(basename "$config" .json)").xml" + + echo "::group::pluginval test: $name" + + # pluginval only runs the first entry of an array config, so reject them + # rather than silently skipping tests. + if [ "$(tr -d '[:space:]' < "$config" | head -c 1)" = "[" ]; then + echo "Array configs aren't supported: use one config per file." + code=1 + else + run_pluginval test ${options[@]+"${options[@]}"} --junit-xml "$report" "$config" + code=$? + fi + + echo "::endgroup::" + + if [ $code -eq 0 ]; then + passed=$((passed + 1)) + rows="$rows| \`$name\` | :white_check_mark: passed |"$'\n' + else + failed=$((failed + 1)) + rows="$rows| \`$name\` | :x: failed |"$'\n' + echo "::error::pluginval acceptance test failed: $name" + fi +done + +if [ $index -eq 0 ]; then + echo "::error::No configs (*.json) found under $config_dir" + exit 1 +fi + +# Gather the diff WAVs written next to failing references, so they can be +# uploaded along with the reports. +find "$config_dir" -type f -name '*-diff.wav' | while IFS= read -r diff; do + mkdir -p "$report_dir/diffs" + cp "$diff" "$report_dir/diffs/$(sanitise "${diff#"$config_dir"/}")" +done + +{ + echo "### pluginval acceptance tests" + echo + echo "| Config | Result |" + echo "| --- | --- |" + printf '%s' "$rows" +} >> "$GITHUB_STEP_SUMMARY" + +{ + echo "passed=$passed" + echo "failed=$failed" + echo "report-dir=$report_dir" +} >> "$GITHUB_OUTPUT" + +echo "pluginval acceptance tests: $passed passed, $failed failed" +[ $failed -eq 0 ] diff --git a/.github/actions/acceptance/action.yml b/.github/actions/acceptance/action.yml new file mode 100644 index 0000000..03f2cce --- /dev/null +++ b/.github/actions/acceptance/action.yml @@ -0,0 +1,51 @@ +name: pluginval acceptance tests +description: Runs pluginval's golden-file acceptance tests for every config in a directory. + +inputs: + config-dir: + description: > + Directory searched recursively for test configs (*.json). The + .wav.json manifests are skipped. + required: true + plugin: + description: > + Plugin path or AU id to test every config against, instead of the + "plugin" in each config. + default: '' + record-missing: + description: Record a missing reference and pass, instead of failing. + default: 'false' + timeout-ms: + description: Fail a test if it takes longer than this (pluginval's default if empty). + default: '' + pluginval-version: + description: The pluginval release to download - a tag (e.g. v2.0.0-beta1) or "latest". + default: v2.0.0-beta1 + pluginval-path: + description: Path to an existing pluginval binary to use instead of downloading one. + default: '' + +outputs: + passed: + description: The number of configs that passed. + value: ${{ steps.run.outputs.passed }} + failed: + description: The number of configs that failed. + value: ${{ steps.run.outputs.failed }} + report-dir: + description: Directory holding a JUnit XML report per config, plus diffs/ with any diff WAVs. + value: ${{ steps.run.outputs.report-dir }} + +runs: + using: composite + steps: + - id: run + shell: bash + env: + INPUT_CONFIG_DIR: ${{ inputs.config-dir }} + INPUT_PLUGIN: ${{ inputs.plugin }} + INPUT_RECORD_MISSING: ${{ inputs.record-missing }} + INPUT_TIMEOUT_MS: ${{ inputs.timeout-ms }} + INPUT_PLUGINVAL_VERSION: ${{ inputs.pluginval-version }} + INPUT_PLUGINVAL_PATH: ${{ inputs.pluginval-path }} + run: bash "$GITHUB_ACTION_PATH/acceptance.sh" diff --git a/.github/actions/common/common.sh b/.github/actions/common/common.sh new file mode 100644 index 0000000..0bdccce --- /dev/null +++ b/.github/actions/common/common.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# Shared helpers for the pluginval composite actions (sourced, not run). +# Kept compatible with bash 3.2, which is what macOS runners ship. + +# Defaults so the scripts can also be run outside of GitHub Actions. +if [ -z "${RUNNER_OS:-}" ]; then + if [ "$(uname -s)" = "Darwin" ]; then RUNNER_OS=macOS + elif [ "$(uname -s)" = "Linux" ]; then RUNNER_OS=Linux + else RUNNER_OS=Windows + fi +fi + +: "${RUNNER_TEMP:=$(mktemp -d)}" +: "${GITHUB_STEP_SUMMARY:=/dev/null}" +: "${GITHUB_OUTPUT:=/dev/null}" + +# Sets PLUGINVAL_BIN to the pluginval binary: INPUT_PLUGINVAL_PATH if given, +# otherwise the INPUT_PLUGINVAL_VERSION release ("latest" or a tag like v2.0.0-beta1) +# downloaded for this runner's OS. +resolve_pluginval() +{ + if [ -n "${INPUT_PLUGINVAL_PATH:-}" ]; then + PLUGINVAL_BIN="$INPUT_PLUGINVAL_PATH" + else + local version="${INPUT_PLUGINVAL_VERSION:-latest}" + local dir="$RUNNER_TEMP/pluginval-$version" + local url="https://github.com/Tracktion/pluginval/releases/download/$version/pluginval_$RUNNER_OS.zip" + + if [ "$version" = "latest" ]; then + url="https://github.com/Tracktion/pluginval/releases/latest/download/pluginval_$RUNNER_OS.zip" + fi + + case "$RUNNER_OS" in + macOS) PLUGINVAL_BIN="$dir/pluginval.app/Contents/MacOS/pluginval" ;; + Windows) PLUGINVAL_BIN="$dir/pluginval.exe" ;; + *) PLUGINVAL_BIN="$dir/pluginval" ;; + esac + + if [ ! -f "$PLUGINVAL_BIN" ]; then + echo "Downloading $url" + mkdir -p "$dir" + + if ! curl -fsSL --retry 3 "$url" -o "$dir/pluginval.zip"; then + echo "::error::Couldn't download pluginval $version for $RUNNER_OS ($url)" + exit 1 + fi + + unzip -q -o "$dir/pluginval.zip" -d "$dir" + chmod +x "$PLUGINVAL_BIN" + fi + fi + + if [ ! -f "$PLUGINVAL_BIN" ]; then + echo "::error::pluginval binary not found: $PLUGINVAL_BIN" + exit 1 + fi +} + +# Runs pluginval with the given arguments. On Linux without a display it is +# wrapped in xvfb-run, as plugins generally need an X server to load. +run_pluginval() +{ + if [ "$RUNNER_OS" = "Linux" ] && [ -z "${DISPLAY:-}" ] && command -v xvfb-run > /dev/null; then + xvfb-run --auto-servernum "$PLUGINVAL_BIN" "$@" + else + "$PLUGINVAL_BIN" "$@" + fi +} + +# Prints a file-name-safe version of the argument. +sanitise() +{ + printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_' +} diff --git a/.github/actions/validate/action.yml b/.github/actions/validate/action.yml new file mode 100644 index 0000000..0eaceb6 --- /dev/null +++ b/.github/actions/validate/action.yml @@ -0,0 +1,48 @@ +name: pluginval validate +description: Validates one or more audio plugins with pluginval. + +inputs: + plugins: + description: > + The plugins to validate, one per line. Each is a path, a glob (e.g. + build/*_artefacts/Release/VST3/*.vst3) or an AU id. + required: true + strictness-level: + description: The strictness level (1-10). + default: '5' + args: + description: > + Extra options for "pluginval validate", split on whitespace (e.g. + "--skip-gui-tests --timeout-ms 60000"). Any option can also be set as an + environment variable on the step. + default: '' + pluginval-version: + description: The pluginval release to download - a tag (e.g. v2.0.0-beta1) or "latest". + default: v2.0.0-beta1 + pluginval-path: + description: Path to an existing pluginval binary to use instead of downloading one. + default: '' + +outputs: + passed: + description: The number of plugins that passed. + value: ${{ steps.run.outputs.passed }} + failed: + description: The number of plugins that failed. + value: ${{ steps.run.outputs.failed }} + report-dir: + description: Directory holding a JUnit XML report per plugin. + value: ${{ steps.run.outputs.report-dir }} + +runs: + using: composite + steps: + - id: run + shell: bash + env: + INPUT_PLUGINS: ${{ inputs.plugins }} + INPUT_STRICTNESS_LEVEL: ${{ inputs.strictness-level }} + INPUT_ARGS: ${{ inputs.args }} + INPUT_PLUGINVAL_VERSION: ${{ inputs.pluginval-version }} + INPUT_PLUGINVAL_PATH: ${{ inputs.pluginval-path }} + run: bash "$GITHUB_ACTION_PATH/validate.sh" diff --git a/.github/actions/validate/validate.sh b/.github/actions/validate/validate.sh new file mode 100644 index 0000000..7ad2841 --- /dev/null +++ b/.github/actions/validate/validate.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Validates each plugin in INPUT_PLUGINS (one per line: a path, a glob or an AU +# id), runs them all, and fails if any failed. See action.yml for the inputs. + +set -u +source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../common/common.sh" + +resolve_pluginval + +report_dir="${INPUT_REPORT_DIR:-$RUNNER_TEMP/pluginval-reports/validate}" +mkdir -p "$report_dir" + +passed=0 +failed=0 +index=0 +rows="" + +# One plugin per line (not comma-separated: AU ids contain commas). Globs are +# left to expand; anything that matches nothing is passed through and fails. +IFS=$'\n' +plugins=( ${INPUT_PLUGINS:-} ) +unset IFS + +for plugin in ${plugins[@]+"${plugins[@]}"}; do + plugin="$(printf '%s' "$plugin" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" + [ -z "$plugin" ] && continue + + index=$((index + 1)) + report="$report_dir/$index-$(sanitise "$(basename "$plugin")").xml" + + echo "::group::pluginval validate: $plugin" + # INPUT_ARGS is deliberately unquoted so it splits into separate arguments. + run_pluginval validate --strictness-level "${INPUT_STRICTNESS_LEVEL:-5}" --junit-xml "$report" ${INPUT_ARGS:-} "$plugin" + code=$? + echo "::endgroup::" + + if [ $code -eq 0 ]; then + passed=$((passed + 1)) + rows="$rows| \`$plugin\` | :white_check_mark: passed |"$'\n' + else + failed=$((failed + 1)) + rows="$rows| \`$plugin\` | :x: failed |"$'\n' + echo "::error::pluginval validation failed: $plugin" + fi +done + +if [ $index -eq 0 ]; then + echo "::error::No plugins to validate (the 'plugins' input is empty)" + exit 1 +fi + +{ + echo "### pluginval validate" + echo + echo "| Plugin | Result |" + echo "| --- | --- |" + printf '%s' "$rows" +} >> "$GITHUB_STEP_SUMMARY" + +{ + echo "passed=$passed" + echo "failed=$failed" + echo "report-dir=$report_dir" +} >> "$GITHUB_OUTPUT" + +echo "pluginval validate: $passed passed, $failed failed" +[ $failed -eq 0 ] diff --git a/.github/workflows/acceptance.yml b/.github/workflows/acceptance.yml index f764864..8109cdf 100644 --- a/.github/workflows/acceptance.yml +++ b/.github/workflows/acceptance.yml @@ -68,3 +68,102 @@ jobs: if: runner.os != 'Linux' working-directory: build run: ctest -C Release --output-on-failure -R pluginval.acceptance + + # The reusable actions in .github/actions, run against the build above + # (pluginval-path) rather than a downloaded release. + - name: Prepare action fixtures + id: fixtures + shell: bash + run: | + case "$RUNNER_OS" in + macOS) echo "pluginval=build/pluginval_artefacts/Release/pluginval.app/Contents/MacOS/pluginval" >> "$GITHUB_OUTPUT" ;; + Windows) echo "pluginval=build/pluginval_artefacts/Release/pluginval.exe" >> "$GITHUB_OUTPUT" ;; + *) echo "pluginval=build/pluginval_artefacts/Release/pluginval" >> "$GITHUB_OUTPUT" ;; + esac + + # Tone generator configs with the plugin removed, so it has to come from + # the action's "plugin" input. One is nested and a reference manifest is + # added, to check the recursion and that manifests aren't run as configs. + generated=build/tests/acceptance/Release + fixtures=build/action-fixtures + mkdir -p $fixtures/good/nested $fixtures/bad $fixtures/missing + sed '/"plugin"/d' $generated/sine-440.json > $fixtures/good/sine-440.json + sed '/"plugin"/d' $generated/square-220.json > $fixtures/good/nested/square-220.json + cp tests/acceptance/refs/sine-440.wav.json $fixtures/good/nested/ + + # A mismatching reference and an (unsupported) array config. + sed -e '/"plugin"/d' -e 's#refs/sine-440.wav#refs/square-220.wav#' $generated/sine-440.json > $fixtures/bad/wrong-ref.json + (echo "["; cat $fixtures/good/sine-440.json; echo "]") > $fixtures/bad/array.json + + # A reference that doesn't exist yet. + sed -e '/"plugin"/d' -e 's#"reference": ".*"#"reference": "recorded.wav"#' $generated/sine-440.json > $fixtures/missing/missing.json + + - name: "Action: validate" + id: validate + uses: ./.github/actions/validate + with: + pluginval-path: ${{ steps.fixtures.outputs.pluginval }} + args: --skip-gui-tests + plugins: build/tests/test_plugins/*/*_artefacts/Release/VST3/*.vst3 + + - name: "Action: validate (missing plugin, expected to fail)" + id: validate-missing + continue-on-error: true + uses: ./.github/actions/validate + with: + pluginval-path: ${{ steps.fixtures.outputs.pluginval }} + args: --skip-gui-tests + plugins: | + build/tests/test_plugins/gain/*_artefacts/Release/VST3/*.vst3 + build/Missing.vst3 + + - name: "Action: acceptance" + id: acceptance + uses: ./.github/actions/acceptance + with: + pluginval-path: ${{ steps.fixtures.outputs.pluginval }} + config-dir: build/action-fixtures/good + plugin: build/tests/test_plugins/tone_generator/pluginval_tone_generator_artefacts/Release/VST3/pluginval Tone Generator.vst3 + + - name: "Action: acceptance (bad configs, expected to fail)" + id: acceptance-bad + continue-on-error: true + uses: ./.github/actions/acceptance + with: + pluginval-path: ${{ steps.fixtures.outputs.pluginval }} + config-dir: build/action-fixtures/bad + plugin: build/tests/test_plugins/tone_generator/pluginval_tone_generator_artefacts/Release/VST3/pluginval Tone Generator.vst3 + + - name: "Action: acceptance (missing reference, expected to fail)" + id: acceptance-missing + continue-on-error: true + uses: ./.github/actions/acceptance + with: + pluginval-path: ${{ steps.fixtures.outputs.pluginval }} + config-dir: build/action-fixtures/missing + plugin: build/tests/test_plugins/tone_generator/pluginval_tone_generator_artefacts/Release/VST3/pluginval Tone Generator.vst3 + + - name: "Action: acceptance (record the missing reference)" + id: acceptance-record + uses: ./.github/actions/acceptance + with: + pluginval-path: ${{ steps.fixtures.outputs.pluginval }} + config-dir: build/action-fixtures/missing + plugin: build/tests/test_plugins/tone_generator/pluginval_tone_generator_artefacts/Release/VST3/pluginval Tone Generator.vst3 + record-missing: true + + - name: Check the action results + shell: bash + run: | + check() { echo "$1: got '$2', expected '$3'"; [ "$2" = "$3" ] || exit 1; } + check "validate passed" "${{ steps.validate.outputs.passed }}" 3 + check "validate failed" "${{ steps.validate.outputs.failed }}" 0 + check "validate-missing outcome" "${{ steps.validate-missing.outcome }}" failure + check "validate-missing passed" "${{ steps.validate-missing.outputs.passed }}" 1 + check "validate-missing failed" "${{ steps.validate-missing.outputs.failed }}" 1 + check "acceptance passed" "${{ steps.acceptance.outputs.passed }}" 2 + check "acceptance failed" "${{ steps.acceptance.outputs.failed }}" 0 + check "acceptance-bad outcome" "${{ steps.acceptance-bad.outcome }}" failure + check "acceptance-bad failed" "${{ steps.acceptance-bad.outputs.failed }}" 2 + check "acceptance-missing outcome" "${{ steps.acceptance-missing.outcome }}" failure + check "acceptance-record passed" "${{ steps.acceptance-record.outputs.passed }}" 1 diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index fbb9dcb..48bdff5 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -218,6 +218,8 @@ jobs: uses: softprops/action-gh-release@v2 with: body_path: CHANGELIST.md + # Tags like v2.0.0-beta1 are pre-releases, so they don't become "latest" + prerelease: ${{ contains(github.ref_name, '-') }} files: | CHANGELIST.md/* pluginval_Linux.zip/* diff --git a/CHANGELIST.md b/CHANGELIST.md index c3f546e..0ae417d 100644 --- a/CHANGELIST.md +++ b/CHANGELIST.md @@ -5,6 +5,8 @@ - Fixed the crash handler so a backtrace is shown again when a plugin crashes, and made it async-signal-safe [#144] - Added `--junit-xml ` and `--json-results ` to both `pluginval validate` and `pluginval test` to write machine-readable reports for CI (also settable as `JUNIT_XML` / `JSON_RESULTS`). A report that can't be written fails the run. See `docs/Adding pluginval to CI.md` - Added acceptance testing: `pluginval test ` renders a plugin with a known state, input (audio/MIDI) and optional playhead, then compares the output against a checked-in reference WAV, exiting `0` on a match and `1` on a mismatch (with a diff WAV). Use `--record-missing` to record new references. See `docs/Acceptance testing.md` +- Added `--plugin ` to `pluginval test` to run a config against a given plugin instead of the one it names (the config's `plugin` can then be omitted) +- Added two GitHub Actions for use from other repositories: `Tracktion/pluginval/.github/actions/validate` (validate a list of plugins) and `.../acceptance` (run every acceptance config in a directory against a built plugin). See `.github/actions/README.md` - Restructured the command line into subcommands: `pluginval validate [options] ` (the default), `pluginval run-tests` and `pluginval strictness-help [level]`. The plugin path is now a positional argument of `validate` - **Deprecated:** the flat flags `--validate `, `--run-tests` and `--strictness-help [level]` still work as aliases (with a one-line notice) but will be removed in a future version. `pluginval ` remains a silent shorthand for `validate` - Replaced the hand-rolled command-line parser with CLI11 and a single JSON-based settings pipeline diff --git a/CLAUDE.md b/CLAUDE.md index e4a4a20..074a158 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -121,6 +121,9 @@ pluginval/ │ ├── acceptance/ # Acceptance self-tests: configs (*.json.in), inputs/, checked-in refs/ WAVs │ ├── mac_tests/ # macOS-specific tests │ └── windows_tests.bat # Windows test scripts +├── .github/ +│ ├── actions/ # Composite actions for other repos: validate/, acceptance/, common/ (see README.md there) +│ └── workflows/ # build.yaml (release build), acceptance.yml (acceptance self-tests + action dogfooding) ├── docs/ # Documentation │ ├── Adding pluginval to CI.md │ ├── Command line options.md @@ -281,7 +284,9 @@ spec: `tests/acceptance/Acceptance testing design.md`; end-user guide: - **CLI**: `pluginval test `. A new `Command::test` is recognised by `settings_parser::dispatch()` (captures the positional config path into `DispatchResult::testConfigPath`, plus `recordMissing` / `timeoutMs` from - `--record-missing` / `--timeout-ms` or `RECORD_MISSING` / `TIMEOUT_MS`), + `--record-missing` / `--timeout-ms` or `RECORD_MISSING` / `TIMEOUT_MS`, and + `testPluginOverride` from `--plugin`, which replaces the config's `plugin` + in `runTestFile()` - so `plugin` is only required when `--plugin` is absent), `isCommandLine()` and `getFooterText()`; `CommandLine.cpp`'s `performCommandLine()` has a `Command::test` branch that starts an `acceptance::TestRunner`, which renders on a **background thread** (message @@ -523,7 +528,26 @@ add_pluginval_tests(MyPluginTarget ) ``` -### GitHub Actions Example +### GitHub Actions + +Two composite actions for other repositories live in `.github/actions/` +(documented in `.github/actions/README.md`): + +- `validate/` - validates a newline-separated list of plugins / globs / AU ids. +- `acceptance/` - runs `pluginval test` for every `*.json` (recursively, skipping + the `*.wav.json` manifests and rejecting array configs) under a directory, + optionally passing one built plugin via `--plugin`. + +Each `action.yml` only maps its inputs to `INPUT_*` environment variables and +runs its sibling bash script (`validate.sh` / `acceptance.sh`); shared helpers +(downloading the release or using `pluginval-path`, the Linux `xvfb-run` wrapper) +are in `common/common.sh`. The scripts must stay bash 3.2 compatible (macOS +runners) and can be run locally by setting the `INPUT_*` variables. They run +every item, write a job-summary table and `passed` / `failed` / `report-dir` +outputs, and exit non-zero if anything failed. `.github/workflows/acceptance.yml` +dogfoods both against the in-repo build, including the expected-failure cases. + +Without the actions: ```yaml - name: Download pluginval @@ -573,6 +597,11 @@ Run internal tests via CLI: 3. Commit: `git commit -am "Version X.Y.Z"` 4. Tag: `git tag -a vX.Y.Z -m "X.Y.Z release"` 5. Push: `git push --tags` +6. Once the release binaries are published, bump the `pluginval-version` default + in both `.github/actions/*/action.yml` (and the README tables) to the new tag + +A tag with a hyphen (e.g. `v2.0.0-beta1`) is published as a pre-release, so it +doesn't become the "latest" release that the download URLs in the docs point at. ## Common Tasks for AI Assistants diff --git a/docs/Acceptance testing.md b/docs/Acceptance testing.md index 018305f..cc3b10f 100644 --- a/docs/Acceptance testing.md +++ b/docs/Acceptance testing.md @@ -46,7 +46,7 @@ JSON keys are `snake_case`. The most useful fields: | Field | Notes | |---|---| | `name` | Labels the result and gives the default reference path. **Required** unless `reference` is given. | -| `plugin` | Path to the plugin (or an AU identifier). **Required.** | +| `plugin` | Path to the plugin (or an AU identifier). **Required**, unless `--plugin` is given on the command line. | | `input.audio` / `input.midi` | Input files to feed it. Omit both for silence (e.g. instruments driven only by MIDI, or generators). The audio file's sample rate must match `sample_rate`. Each file channel feeds the plugin input channel with the same index: a mono file feeds channel 0 only (it isn't duplicated), and extra channels on either side are silent / ignored. MIDI meta events (tempo, track names) aren't sent to the plugin. | | `state.parameters` | A map of parameter → **normalised** value (`0` to `1`), applied before rendering, **in file order**. A key of digits only is a parameter index; otherwise it matches the format's parameter ID, then the display name (case-insensitively). | | `state.file` | A binary `getStateInformation` blob to restore first (e.g. a captured preset). Applied before `state.parameters`. | @@ -89,6 +89,9 @@ Any other option is an error. `RECORD_MISSING=1`. - `--timeout-ms ` - fail (exit `1`) if the run takes longer than this (default `30000`, `-1` to never time out). Also settable as `TIMEOUT_MS`. +- `--plugin ` - test this plugin path (or AU identifier) instead of the + config's `plugin`, which can then be left out. This lets one set of configs be + run against whichever binary was just built (e.g. both the VST3 and the AU). - `--junit-xml ` - write a JUnit XML report, for CI systems that display test results. Also settable as `JUNIT_XML`. - `--json-results ` - write a JSON report (the same result that is printed @@ -97,6 +100,18 @@ Any other option is an error. See [Adding pluginval to CI](Adding%20pluginval%20to%20CI.md#machine-readable-reports) for the report formats. +On GitHub Actions, the `acceptance` action runs every config in a directory +(recursively) against a built plugin: + +```yaml +- uses: Tracktion/pluginval/.github/actions/acceptance@v2 + with: + config-dir: tests/acceptance + plugin: build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3 +``` + +See the [actions README](../.github/actions/README.md) for its inputs and outputs. + The render runs on a background thread, as a host's audio thread would, leaving the message thread free for the plugin. diff --git a/docs/Adding pluginval to CI.md b/docs/Adding pluginval to CI.md index 43a63e4..22b1c85 100644 --- a/docs/Adding pluginval to CI.md +++ b/docs/Adding pluginval to CI.md @@ -9,6 +9,21 @@ There a couple of different ways this can be done but the simplest is as follows 4. If validation fails, fail your build 5. If validation passes, retain the generated log files ready to submit to "Verified by pluginval" +### GitHub Actions +On GitHub Actions the simplest route is the `validate` action, which downloads pluginval, validates each plugin and fails the job if any fail: + +```yaml +- uses: Tracktion/pluginval/.github/actions/validate@v2 + with: + plugins: | + build/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3 + build/MyPlugin_artefacts/Release/AU/MyPlugin.component + strictness-level: 5 + args: --skip-gui-tests +``` + +There is also an `acceptance` action for [acceptance tests](Acceptance%20testing.md). See the [actions README](../.github/actions/README.md) for all the inputs and outputs. + ### Examples In these examples the following is assumed: - macOS/Linux, curl is installed diff --git a/docs/Command line options.md b/docs/Command line options.md index 71e2128..90442aa 100644 --- a/docs/Command line options.md +++ b/docs/Command line options.md @@ -20,6 +20,8 @@ COMMANDS: (default 30000, -1 = never). --junit-xml Write a JUnit XML report. --json-results Write a JSON report. + --plugin Test this plugin path or AU id + instead of the config's. Any other option is an error. The flat flags --validate , --run-tests and --strictness-help [level] diff --git a/source/CommandLine.cpp b/source/CommandLine.cpp index 9736bd1..1b92bbf 100644 --- a/source/CommandLine.cpp +++ b/source/CommandLine.cpp @@ -231,6 +231,7 @@ void performCommandLine (CommandLineValidator& validator, const juce::String& co acceptance::RunOptions options; options.recordMissing = routed.recordMissing; options.timeoutMs = routed.timeoutMs; + options.pluginOverride = routed.testPluginOverride.toStdString(); const auto toReportFile = [] (const juce::String& path) { diff --git a/source/CommandLineTests.cpp b/source/CommandLineTests.cpp index bb5fed9..869fe2c 100644 --- a/source/CommandLineTests.cpp +++ b/source/CommandLineTests.cpp @@ -543,6 +543,29 @@ struct CommandLineTests : public juce::UnitTest } } + beginTest ("Subcommand: test plugin override (--plugin)"); + { + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test --plugin build/P.vst3 config.json")); + expect (d.error.isEmpty()); + expectEquals (d.testPluginOverride, juce::String ("build/P.vst3")); + expectEquals (d.testConfigPath, juce::String ("config.json")); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json")); + expect (d.testPluginOverride.isEmpty()); + } + { + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json --plugin")); + expect (d.error.contains ("--plugin")); + } + { + // The following option isn't taken as the plugin. + const auto d = settings_parser::dispatch (settings_parser::tokenise ("test config.json --plugin --record-missing")); + expect (d.error.contains ("--plugin")); + } + } + beginTest ("Subcommand: test report options (--junit-xml, --json-results)"); { { @@ -840,7 +863,7 @@ struct CommandLineTests : public juce::UnitTest expect (! rejects (R"({ "name": "t", "plugin": "P.vst3" })")); expect (! rejects (R"({ "reference": "r.wav", "plugin": "P.vst3" })"), "reference alone names the test"); - expect (rejects (R"({ "name": "t" })"), "plugin is required"); + expect (! rejects (R"({ "name": "t" })"), "plugin can come from --plugin instead"); expect (rejects (R"({ "plugin": "P.vst3" })"), "name or reference is required"); expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "comparison": {} })"), "empty comparison"); expect (rejects (R"({ "name": "t", "plugin": "P.vst3", "comparison": 1 })"), "non-object comparison"); diff --git a/source/SettingsParser.cpp b/source/SettingsParser.cpp index 999a6a6..d8105eb 100644 --- a/source/SettingsParser.cpp +++ b/source/SettingsParser.cpp @@ -137,6 +137,8 @@ R"(Commands: (default 30000, -1 to never time out). --junit-xml Write a JUnit XML report. --json-results Write a JSON report. + --plugin Test this plugin path or AU id instead + of the one in the config. The flat flags --validate , --run-tests and --strictness-help [level] are deprecated aliases for the commands above and will be removed in a future version. @@ -424,6 +426,18 @@ Precedence (lowest to highest): defaults, environment variables, --config, comma (token == "--junit-xml" ? result.junitXmlPath : result.jsonResultsPath) = value; } + else if (token == "--plugin") + { + const auto value = i + 1 < tokensIn.size() ? tokensIn.getReference (++i) : juce::String(); + + if (value.isEmpty() || value.startsWith ("--")) + { + result.error = "--plugin needs a plugin path or AU id"; + break; + } + + result.testPluginOverride = value; + } else if (token.startsWith ("-")) { result.error = "unknown option for test: " + token; diff --git a/source/SettingsParser.h b/source/SettingsParser.h index 934b0c0..6896050 100644 --- a/source/SettingsParser.h +++ b/source/SettingsParser.h @@ -85,6 +85,7 @@ namespace settings_parser juce::int64 timeoutMs = 30000; /**< test: abort the run after this long (-1 to never time out). */ juce::String junitXmlPath; /**< test: file to write a JUnit XML report to. */ juce::String jsonResultsPath; /**< test: file to write a JSON report to. */ + juce::String testPluginOverride; /**< test: plugin path or AU id to use instead of the config's. */ juce::String error; /**< test: an invalid option / argument, to report instead of running. */ }; diff --git a/source/acceptance/AcceptanceTest.cpp b/source/acceptance/AcceptanceTest.cpp index 100cdfb..9b8e2b2 100644 --- a/source/acceptance/AcceptanceTest.cpp +++ b/source/acceptance/AcceptanceTest.cpp @@ -629,8 +629,16 @@ int runTestFile (const juce::File& configFile, const RunOptions& options) if (configs.size() > 1) std::cout << "Note: " << configs.size() << " configs found; v1 runs the first only." << std::endl; + auto& config = configs.front(); + + if (! options.pluginOverride.empty()) + config.plugin = options.pluginOverride; + + if (config.plugin.empty()) + return finish (TestResult::makeError (config.getName(), "plugin is required (in the config, or via --plugin)")); + const auto startTime = juce::Time::getMillisecondCounterHiRes(); - auto result = runTest (configs.front(), options); + auto result = runTest (config, options); result.durationSeconds = (juce::Time::getMillisecondCounterHiRes() - startTime) / 1000.0; return finish (result); } diff --git a/source/acceptance/AcceptanceTest.h b/source/acceptance/AcceptanceTest.h index a8d1d93..8dcb1f1 100644 --- a/source/acceptance/AcceptanceTest.h +++ b/source/acceptance/AcceptanceTest.h @@ -54,6 +54,7 @@ struct RunOptions juce::int64 timeoutMs = 30000; /**< Abort the whole run after this long. <= 0 never times out. */ juce::File junitXmlFile; /**< File to write a JUnit XML report to (none if default). */ juce::File jsonResultsFile; /**< File to write a JSON report to (none if default). */ + std::string pluginOverride; /**< Plugin path or AU id to test instead of the config's (none if empty). */ }; //============================================================================== diff --git a/source/acceptance/TestConfig.cpp b/source/acceptance/TestConfig.cpp index bdb25d0..3f13495 100644 --- a/source/acceptance/TestConfig.cpp +++ b/source/acceptance/TestConfig.cpp @@ -134,10 +134,8 @@ void from_json (const nlohmann::ordered_json& j, TestConfig& c) c.comparison = *comp; } - // Validation. - if (c.plugin.empty()) - fail ("plugin is required"); - + // Validation. (The plugin may instead come from --plugin, so it's checked + // by the runner once any override has been applied.) if (c.name.empty() && c.reference.empty()) fail ("name or reference is required"); diff --git a/source/acceptance/TestConfig.h b/source/acceptance/TestConfig.h index 556a582..a6b2e01 100644 --- a/source/acceptance/TestConfig.h +++ b/source/acceptance/TestConfig.h @@ -53,7 +53,7 @@ struct TestConfig }; std::string name; /**< Labels results; derives the default reference path. */ - std::string plugin; /**< Plugin path or AU id. Required. */ + std::string plugin; /**< Plugin path or AU id. Required unless --plugin is given. */ std::string inputAudio; /**< input.audio: path to an input audio file, or empty for silence. */ std::string inputMidi; /**< input.midi: path to a .mid file, or empty for none. */ std::string reference; /**< The golden file. Empty -> derived from name. */ diff --git a/tests/acceptance/Acceptance testing design.md b/tests/acceptance/Acceptance testing design.md index 836a805..27ee2da 100644 --- a/tests/acceptance/Acceptance testing design.md +++ b/tests/acceptance/Acceptance testing design.md @@ -140,7 +140,7 @@ member names verbatim). Missing keys still fall back to the defaults. | Field | Type | Default | Notes | |---|---|---|---| | `name` | string | reference's basename | Used to derive the default reference path and to label results. `name` or `reference` is required. | -| `plugin` | string | (required) | Absolute/relative path, or an AU identifier string. Relative paths resolve against the working directory. | +| `plugin` | string | (required) | Absolute/relative path, or an AU identifier string. Relative paths resolve against the working directory. `--plugin ` on the command line overrides it, in which case it may be omitted. | | `input.audio` | string | none | Path to an input audio file (WAV/AIFF/FLAC via `AudioFormatManager`). Used for effects. Its sample rate must equal `sample_rate`. File channel N feeds plugin input channel N; a mono file feeds channel 0 only (deliberately not duplicated). | | `input.midi` | string | none | Path to a `.mid` file (`juce::MidiFile`). Used for instruments. Meta events are skipped. | | `reference` | string | `/.wav` | The golden file. Absent -> record mode; present -> compare mode. |