TESTING: Make the test failures fatal and resolve all of them - #1371
Draft
ACSimon33 wants to merge 10 commits into
Draft
TESTING: Make the test failures fatal and resolve all of them#1371ACSimon33 wants to merge 10 commits into
ACSimon33 wants to merge 10 commits into
Conversation
…iles are now differentiated between the jobs.
❌ 34 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
The merge kept both sides of the LAPACK_Test_Summary block, so nothing configured at all: add_test refused the second, duplicate, test name. Keep this branch's copy, which passes --fail-on-error and the job id, and take the mention of LAPACKE from master's comment. Moving the input check into the helper also dropped the rule that a BLAS driver reading an input file must not have its stdout redirected: it takes the name of its summary file from that input and opens it on unit 6 itself, so the redirect opened the one file twice and xBLAT2 died on the second open. Only xBLAT1, which has no input, needs it. Also drop a reference to the preprocessor macro the error-exit tests no longer use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ACSimon33
force-pushed
the
fatal_test_failures
branch
from
September 9, 2026 14:47
e1e72bc to
637aa38
Compare
nagfor rejects the SQRT( -ONE ) that creates a NaN, and the LAPACKE test build regenerates the ?errcxx drivers without the -Onopropagate that TESTING/LIN applies, so every nagfor job has failed to build since the LAPACKE LIN tests were merged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every STOP in the test drivers was either bare or carried a character stop code, and both exit with status 0. That is not an accident of the implementation: the standard recommends the exit status be taken from an integer stop code, and recommends zero when the stop code is of type character or absent. So a driver that abandoned its run - an unrecognised subprogram name, an input file it could not parse, a failed ALLOCATE, or its own reference arithmetic evaluating wrongly - reported success, and ctest and lapack_testing.py, which both do check the status, saw a passing test. The 177 fatal paths now use STOP 1. A digit string as the stop code is FORTRAN 77, and STOP keeps normal termination, so all units are still flushed and closed; ERROR STOP would need Fortran 2008 and only recommends flushing. The 24 STOPs that end a program normally are left as they are. The ALLOCATE checks printed their message as the stop code, which sends it to standard error where lapack_testing.py never sees it. They now write it to the output unit before stopping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--fail-if-empty only looked at the grand total, so one driver that wrote its header and then ran nothing passed unnoticed as long as its siblings kept the total above zero. That is the shape of a driver whose input file has stopped lining up with what it reads: the run is not an error by any measure the script applies, the tests simply are not there. Report every analyzed output file whose test count is zero, and let --fail-if-empty fail on it as well. Measured over a clean build, the smallest per-file count is 6 tests and none is zero, so this does not fire on a healthy run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
langou
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The test suite could not fail. A driver that found a numerical error printed it and exited with status 0, and every fatal
STOPwas either bare or carried a character stop code, which also exits with 0 — soctestandlapack_testing.py, which both check the status, saw a passing run. This branch closes those paths and fixes what the CI then reported.Merge this last: the numerical failures it exposes are fixed on their own branches (see table below).
Failures now fail
LAPACK_Test_Summaryruns with--fail-on-error, so a numerical failure or error reported by any driver fails the run.STOPs becomeSTOP 1, across 45 files inBLAS/TESTING,CBLAS/testing,TESTING/EIGandTESTING/LIN.ALLOCATEchecks passed their message as the stop code, which goes to stderr where the script never sees it; they now write it to the output unit.Reports name their job
lapack_testing.py --junit-job NAMEprefixes the JUnit suite (job / suite) and class (job.class) names, exposed to CMake asLAPACK_TESTING_JUNIT_JOB_ID. Codecov keys a test case on those names alone, so the reports of ~40 jobs collapsed onto one another, and a failure did not say where it came from.Error-exit tests
XERBLAcannot replace the library's, so the first deliberate illegal argument killed the driver before any test ran (CBLAS already skips itsxerblatests for this).LAPACK_SKIP_ERROR_EXIT_TESTSplus the newlapack_test_input()feed such a build a copy of the input withTSTERRoff, written wherelapack_testing.pylooks for it so a hand-run driver uses the same inputctestdoes. This could be fixed byREGISTER_XERBLA- override the error-handler without linker stuff #1407.?ec.inhad noTSTERRline at all —?chkeehard-coded.TRUE.. It is now read from the file, the line is added tosec.in/dec.in/cec.in/zec.in, and the input-format docs in[sdcz]chkee.Fare updated.The numerical failures, fixed elsewhere
macos-26-nagfor,sec/dec; closes #598ubuntu-26.04-gfortran, the 803-rowcnep/ced/znep/zedbaselineSTFSM1 of 7776, every x86-64 gfortran job; closes #679SHS/CHS1 of 2016, ARM gfortran and armflangubuntu-26.04-arm-flang,ubuntu-24.04-arm-armflang; item 3 of #732Earlier test failures cleared on the way here; all merged: #1337, #1338, #1340, #1342, #1343, #1344, #1346, #1347, #1355.