🤖 Generated by the Agentic Engineer
Evidence
Found while mutation-testing the rules in #4526. When a policy contains a variable Kyverno cannot validate, kyverno test (1.19.1) marks every row of every fixture that loads it Skip with the reason Invalid Policy, counts those rows as passed, and exits 0. scripts/validate-kyverno-fixture-evaluation.sh then reports that every fixture evaluates every rule it names and also exits 0.
Reproduced on a scratch copy of main's tree: one condition key in restrict-github-team-management replaced by a bare JSON literal inside the variable braces. kyverno apply on the same file says the policy "contains invalid variables" and skips it; kyverno test on its fixture directory reports all rows passed; the validator exits 0.
Problem and audience
The validator exists so that a fixture cannot stay green while its rule judges nothing. A policy that fails to load is the largest version of that: none of its rules judged anything, and the suite is green. The audience is whoever changes a policy and relies on the suite: a typo in an expression would read as a full pass here and only surface when the cluster rejects the policy at deploy.
Expected behaviour
A fixture run in which any row was skipped because its policy is invalid fails, naming the policy.
Acceptance criteria
Evidence
Found while mutation-testing the rules in #4526. When a policy contains a variable Kyverno cannot validate,
kyverno test(1.19.1) marks every row of every fixture that loads itSkipwith the reasonInvalid Policy, counts those rows as passed, and exits 0.scripts/validate-kyverno-fixture-evaluation.shthen reports that every fixture evaluates every rule it names and also exits 0.Reproduced on a scratch copy of
main's tree: one condition key inrestrict-github-team-managementreplaced by a bare JSON literal inside the variable braces.kyverno applyon the same file says the policy "contains invalid variables" and skips it;kyverno teston its fixture directory reports all rows passed; the validator exits 0.Problem and audience
The validator exists so that a fixture cannot stay green while its rule judges nothing. A policy that fails to load is the largest version of that: none of its rules judged anything, and the suite is green. The audience is whoever changes a policy and relies on the suite: a typo in an expression would read as a full pass here and only surface when the cluster rejects the policy at deploy.
Expected behaviour
A fixture run in which any row was skipped because its policy is invalid fails, naming the policy.
Acceptance criteria
kyverno testreports a row with the reasonInvalid Policy, whatever result that row declared.