Skip to content

Add --junit-xml and --json-results report output to validate and test - #192

Merged
drowaudio merged 2 commits into
v2from
feature/report-output
Oct 13, 2026
Merged

drowaudio merged 2 commits into
v2from
feature/report-output

Conversation

@drowaudio

Copy link
Copy Markdown
Contributor

Summary

pluginval validate and pluginval test can now write machine-readable reports, for CI systems that display per-test results and for scripts.

Option Env var Effect
--junit-xml <file> JUNIT_XML Write a JUnit XML report
--json-results <file> JSON_RESULTS Write a JSON report

Both are off by default. Relative paths resolve against the working directory and parent directories are created. The text output, log files and test's JSON on stdout are unchanged.

How it works

  • New source/ReportWriter.h/.cpp (report_writer): one neutral Report model, toJUnitXml(), toJson() and write(). Each command converts its own results into it.
  • validate: the options are ordinary settings, so env vars, --config and the child-process handoff work. The report is written in runTests() in Validator.cpp, one test case per pluginval test.
  • test: the options are parsed in settings_parser::dispatch(). The existing result JSON is included in the JSON report under details.
  • A placeholder error report is written before a run and replaced on completion, so a crash never leaves a stale or missing report. On a timeout, the thread that ends the process writes a "timed out" report last, so the report always agrees with the exit code.
  • A report that was requested but can't be written fails the run (exit 1).

Testing

  • New unit tests in CommandLineTests.cpp for option parsing (CLI, env, --config, child round-trip) and the writer (XML parsed back, escaping, JSON summary, file output).
  • The five pluginval.acceptance.* CTest cases now pass both options, so the write path runs in the acceptance workflow on all three platforms.
  • On macOS (Debug): pluginval run-tests passes and the acceptance CTest run passes 5/5. Checked by hand for both commands: success, failure, error, timeout, env vars and an unwritable report path; reports validate with xmllint / jq and agree with the exit code.
  • Not built or run locally on Windows or Linux.

Docs

docs/Adding pluginval to CI.md (new "Machine-readable reports" section), docs/Command line options.md, docs/Acceptance testing.md, CHANGELIST.md and CLAUDE.md.

Both commands can now write machine-readable reports through a shared report_writer. A placeholder error report is written before a run and replaced on completion, and a timeout writes a timed-out report, so a report always agrees with the exit code. A report that can't be written fails the run.
@drowaudio
drowaudio merged commit f257fb4 into v2 Oct 13, 2026
13 checks passed
@drowaudio
drowaudio deleted the feature/report-output branch October 13, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant