Skip to content

Add a discrete parameter state restoration test and fix false failures in the existing one - #191

Open
drowaudio wants to merge 5 commits into
v2from
dev/discrete-params-validation-b488be
Open

drowaudio wants to merge 5 commits into
v2from
dev/discrete-params-validation-b488be

Conversation

@drowaudio

@drowaudio drowaudio commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Follows up the forum thread Pluginval can set an AudioParameterBools to odd values. The sporadic "Plugin state restoration" failures turn out to be two separate problems, one in plugins and one in pluginval's own test.

1. New test: "Discrete parameter state restoration" (strictness 8)

For each discrete parameter: set a random value, save state, then restore that state twice - once untouched and once after setting a different value - and expect the same value both times.

  • It does not guess how the plugin maps a normalised value to a step, so pluginval keeps sending arbitrary floats.
  • Both reads happen after a state restore, so the host's cached value does not interfere.
  • It fails for parameters that keep the raw normalised value from the host but save/restore the step. juce::AudioParameterBool::setValue does this in JUCE 9.0.3 (Choice, Int and Float snap).

tests/test_plugins/discrete_params/ is a minimal repro plugin (3 bools, 2 choices, VST3 + AU on macOS), built with PLUGINVAL_BUILD_TEST_PLUGINS. It is not wired into CTest because it is expected to fail at level 8 until the JUCE bool is fixed.

2. Fix: false failures in "Plugin state restoration"

Hosted VST3 and AU parameters return the last value the host sent until a state is restored. The first parameter's originalValue could therefore be whatever a previous test sent, not the plugin's value, which is why only the first bool or choice failed for some people. The test now restores the original state once before reading any values.

3. Only send 0 or 1 to two-step parameters below strictness 8

Every test that randomises parameter values now goes through getRandomParameterValue(). Below level 8 it returns 0 or 1 for two-step (on/off) parameters; at 8 and above, and for every other parameter at any level, it returns an arbitrary float as before.

  • 0 and 1 are the two values every mapping agrees on, so this involves no guessing, unlike snapping a multi-step parameter.
  • Two-step is isBoolean() || getNumSteps() == 2, because JUCE's hosted VST3 parameters never report isBoolean().
  • Levels 6 and 7 no longer exercise in-between values on on/off parameters. Level 8+ still does.

4. Fix: the binary state check in "Plugin state restoration" never ran

The test documents an exact binary state comparison at strictness 8 and above, but the condition read the test's own level (6) instead of the run's, so it has never executed. It now uses ut.getOptions().strictnessLevel.

  • With the fix the test makes 6 checks on the repro plugin at level 8, up from 5.
  • No "Returned state differs" failures on the four in-repo test plugins (VST3, levels 7, 8 and 10) or on six of Apple's built-in Audio Units at level 8.
  • No third-party plugins were tried. A plugin that writes anything non-deterministic into its state will start failing at level 8 and above.

Results

Against the repro plugin on stock JUCE 9.0.3, 12 seeds x 5 randomised repeats, VST3 and AU:

Strictness Result
7 0 failures in 12 of 12 seeds, both formats
8 fails in 12 of 12 seeds, both formats, on bools only, in both the new test and "Plugin state restoration"

Before change 3, with 20 randomised repeats at strictness 10:

Plugin's JUCE New test Existing test, before fix 2 Existing test, after fix 2
Stock 9.0.3 fails on all 3 bools fails 9-17 of 20 per bool still fails on all 3 bools (VST3, 12 seeds)
Bool patched to snap 0 failures Bool 1 only: 13 (VST3), 7 (AU) 0 failures, VST3 and AU

With the choices declared first and the bool patched, the unfixed existing test failed on Choice 1 in 8 of 12 seeds (VST3).

Notes for review

  • Until JUCE is fixed, a JUCE plugin with an AudioParameterBool fails at strictness 8 and above. Both thresholds are open to discussion.
  • That includes JUCE's own DSPModulePluginDemo, which CI validates at strictness 10: the new test failed on 12 of its bool parameters. CI now passes --disabled-tests "Discrete parameter state restoration" for that plugin only (VST3 and AU), to be removed once JUCE is fixed.
  • The JUCE-side fix is value = convertFrom0to1 (newValue) in AudioParameterBool::setValue, matching Choice/Int/Float. That is not part of this PR.
  • Change 4 enables a check that has never run, so it may newly fail third-party plugins at level 8 and above.
  • Only tested on macOS.

Refs #188

Checks that a discrete parameter has the same value after restoring a
state, whatever value it held beforehand. This fails for parameters that
keep the raw normalised value from the host instead of mapping it to a
step, e.g. juce::AudioParameterBool.

See https://forum.juce.com/t/pluginval-can-set-an-audioparameterbools-to-odd-values/50971
Hosted VST3 and AU parameters return the last value sent by the host
until a state is restored. The first parameter's original value could
therefore be whatever a previous test sent, not the plugin's own value.
Restore the original state once before reading any values.
Some plugins keep in-between values for on/off parameters rather than
mapping them to a step (e.g. juce::AudioParameterBool), which made the
plugin state restoration test fail at level 6. Arbitrary values are
still sent to these parameters at level 8 and above.

Hosted VST3 parameters don't report isBoolean, so two-step parameters
are identified by their number of steps as well.
The check is documented as running at strictness level 8 and above but
compared the test's own level (6) rather than the run's, so it was
never executed.
juce::AudioParameterBool::setValue keeps the raw value it is given, so
the JUCE demo fails the new test at strictness level 8 and above. Skip
that one test for the demo until it is fixed in JUCE.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant