Repository navigation
Test fails but doesn't fail as expected (tests/interp/m98m99/12-M99-endless-main-program changed output) #4537
Description
Activity
BTW, a test that is supposed to fail (using
xfail) still needs to test whether the failure mode is exactly the failure expected. That makes things very difficult. Tests should test the result and make the decision on that result. If the result changes, then the test should always be rejected as not passing.The test in question,
tests/interp/m98m99/12-M99-endless-main-program/is the only one using thexfailmarker. It would probably be prudent to remove that functionality completely. The test can do logic negation when necessary. No need for the runtests script to make things complicated by negating negations by negating the negative not negated by negation... (if you still can follow my line of thought)Reacted by alex-pres- added a commit that references this issue
on Sep 15, 2026
The test in
tests/interp/m98m99/12-M99-endless-main-programchanged after @grandixximo #4441 merge we get the output:The real problem is that the test is marked
xfailand that means that it is supposed to fail. However, it has not failed for a very long time and we've all gotten used to the message*** /home/runner/work/linuxcnc/linuxcnc/tests/interp/m98m99/12-M99-endless-main-program: XPASS: Passed, but was expected to fail.Now the output changed and nobody caught it because CI lets it pass. We told it to let it pass...
Either the program marked
xfaildoes fail and only passes if it fails. Or, we let it pass and it must always pass. This intermediate state we have now is the same as what happened in #4397 and it will happen again and again as long as we let non-failing tests that should fail pass.The other solution is to disable or remove this test if it does not serve any proper purpose.