[Tests] Lapack tests on windows

[Neumann-A]

Short story. I am currently improving Fortran support in vcpkg and the question is which Fortran compiler to choose. The current implementation uses gfortran and a MinGW toolchain which seems to make static builds on Windows impossible (unless you go full MinGW which isn't the aim).
As alternatives I tried:
ifort and flang-classic and got the following test results:

gfortran:x64-windows-static (using internal BLAS)

debug:
The following tests FAILED:
         46 - LAPACK-xlintstc_ctest_in (Failed)
         47 - LAPACK-xlintstrfc_ctest_rfp_in (Failed)
         48 - LAPACK-xeigtstc_nep_in (Failed)
         49 - LAPACK-xeigtstc_sep_in (Failed)
         50 - LAPACK-xeigtstc_se2_in (Failed)
         51 - LAPACK-xeigtstc_svd_in (Failed)
         52 - LAPACK-xeigtstc_cec_in (Failed)
         53 - LAPACK-xeigtstc_ced_in (Failed)
         55 - LAPACK-xeigtstc_cgd_in (Failed)
         56 - LAPACK-xeigtstc_csb_in (Failed)
         57 - LAPACK-xeigtstc_csg_in (Failed)
         65 - LAPACK-xeigtstc_gsv_in (Failed)
         68 - LAPACK-xlintstz_ztest_in (Failed)
         69 - LAPACK-xlintstrfz_ztest_rfp_in (Failed)

                        -->   LAPACK TESTING SUMMARY  <--
                Processing LAPACK Testing output found in the TESTING directory
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1296891         1       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1305489         0       (0.000%)        0       (0.000%)
COMPLEX                 17208           0       (0.000%)        0       (0.000%)
COMPLEX16               325563          0       (0.000%)        0       (0.000%)

--> ALL PRECISIONS      2945151         1       (0.000%)        0       (0.000%)

release:
The following tests FAILED:
         68 - LAPACK-xeigtstc_csb_in (Failed) <- Due to file access

                        -->   LAPACK TESTING SUMMARY  <--
                Processing LAPACK Testing output found in the TESTING directory
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    39516           0       (0.000%)        16      (0.040%)
DOUBLE PRECISION        39516           0       (0.000%)        17      (0.043%)
COMPLEX                 39516           0       (0.000%)        15      (0.038%)
COMPLEX16               39516           0       (0.000%)        17      (0.043%)

--> ALL PRECISIONS      158064          0       (0.000%)        65      (0.041%)

gfortran:x64-windows (using OpenBLAS; nb test run seems strange)

debug:
                        -->   LAPACK TESTING SUMMARY  <--
                Processing LAPACK Testing output found in the TESTING directory
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    39516           0       (0.000%)        16      (0.040%)                  <- what are these and why is the number so low?
DOUBLE PRECISION        39516           0       (0.000%)        17      (0.043%)
COMPLEX                 39516           0       (0.000%)        16      (0.040%)
COMPLEX16               39516           0       (0.000%)        17      (0.043%)

--> ALL PRECISIONS      158064          0       (0.000%)        66      (0.042%)
release:
                        -->   LAPACK TESTING SUMMARY  <--
                Processing LAPACK Testing output found in the TESTING directory
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    39516           0       (0.000%)        16      (0.040%)
DOUBLE PRECISION        39516           0       (0.000%)        17      (0.043%)
COMPLEX                 39516           0       (0.000%)        16      (0.040%)
COMPLEX16               39516           0       (0.000%)        17      (0.043%)

--> ALL PRECISIONS      158064          0       (0.000%)        66      (0.042%)

flang:x64-windows-static (release not working due to missing reentrant IO feature in flang-classic?)

The following tests FAILED:
         46 - LAPACK-xlintstc_ctest_in (Failed)
         47 - LAPACK-xlintstrfc_ctest_rfp_in (Failed)
         48 - LAPACK-xeigtstc_nep_in (Failed)
         49 - LAPACK-xeigtstc_sep_in (Failed)
         50 - LAPACK-xeigtstc_se2_in (Failed)
         51 - LAPACK-xeigtstc_svd_in (Failed)
         52 - LAPACK-xeigtstc_cec_in (Failed)
         53 - LAPACK-xeigtstc_ced_in (Failed)
         56 - LAPACK-xeigtstc_csb_in (Failed)
         57 - LAPACK-xeigtstc_csg_in (Failed)
         65 - LAPACK-xeigtstc_gsv_in (Failed)
         68 - LAPACK-xlintstz_ztest_in (Failed)
         69 - LAPACK-xlintstrfz_ztest_rfp_in (Failed)

                        -->   LAPACK TESTING SUMMARY  <--
                Processing LAPACK Testing output found in the TESTING directory
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1296891         1       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1305489         0       (0.000%)        0       (0.000%)
COMPLEX                 22690           355     (1.565%)        0       (0.000%)
COMPLEX16               325563          0       (0.000%)        0       (0.000%)

--> ALL PRECISIONS      2950633         356     (0.012%)        0       (0.000%)

Release:
Test ERROR:
FIO-F-235/formatted write/unit=6/edit descriptor does not match item type.
 File name = 'stdout ',    formatted, sequential access   record = 7
 In source file E:\b\lapack-reference\src\v3.10.1-882cc33780\TESTING\EIG\zchkee.F, at line number 1312

CMake Error at E:/b/lapack-reference/src/v3.10.1-882cc33780/TESTING/runtest.cmake:36 (message):
  Test E:/b/lapack-reference/x64-windows-static-rel/bin/xeigtstz.exe returned
  127

ifort:x64-windows

Debug
The following tests FAILED:
         46 - LAPACK-xlintstc_ctest_in (Failed)
         47 - LAPACK-xlintstrfc_ctest_rfp_in (Failed)
         48 - LAPACK-xeigtstc_nep_in (Failed)
         49 - LAPACK-xeigtstc_sep_in (Failed)
         50 - LAPACK-xeigtstc_se2_in (Failed)
         52 - LAPACK-xeigtstc_cec_in (Failed)
         53 - LAPACK-xeigtstc_ced_in (Failed)
         55 - LAPACK-xeigtstc_cgd_in (Failed)
         56 - LAPACK-xeigtstc_csb_in (Failed)
         57 - LAPACK-xeigtstc_csg_in (Failed)
         65 - LAPACK-xeigtstc_gsv_in (Failed)
         68 - LAPACK-xlintstz_ztest_in (Failed)
         69 - LAPACK-xlintstrfz_ztest_rfp_in (Failed)

                        -->   LAPACK TESTING SUMMARY  <--
                Processing LAPACK Testing output found in the TESTING directory
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1310091         1       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1318689         0       (0.000%)        0       (0.000%)
COMPLEX                 46759           0       (0.000%)        5       (0.011%)        <--- broken Tests? I run into a lot of un-init exceptions
COMPLEX16               438860          0       (0.000%)        0       (0.000%)

--> ALL PRECISIONS      3114399         1       (0.000%)        5       (0.000%)

release
The following tests FAILED:
         46 - LAPACK-xlintstc_ctest_in (Failed)
         47 - LAPACK-xlintstrfc_ctest_rfp_in (Failed)
         49 - LAPACK-xeigtstc_sep_in (Failed)
         50 - LAPACK-xeigtstc_se2_in (Failed)
         51 - LAPACK-xeigtstc_svd_in (Failed)
         52 - LAPACK-xeigtstc_cec_in (Failed)
         53 - LAPACK-xeigtstc_ced_in (Failed)
         55 - LAPACK-xeigtstc_cgd_in (Failed)
         56 - LAPACK-xeigtstc_csb_in (Failed)
         57 - LAPACK-xeigtstc_csg_in (Failed)

                        -->   LAPACK TESTING SUMMARY  <--
                Processing LAPACK Testing output found in the TESTING directory
SUMMARY                 nb test run     numerical error         other error
================        ===========     =================       ================
REAL                    1310091         1       (0.000%)        0       (0.000%)
DOUBLE PRECISION        1318689         0       (0.000%)        0       (0.000%)
COMPLEX                 30569           359     (1.174%)        0       (0.000%) <--- release builds don't throw the un-init exceptions. 
COMPLEX16               778430          0       (0.000%)        0       (0.000%)

--> ALL PRECISIONS      3437779         360     (0.010%)        0       (0.000%)

From the test results ifort seems to be the better solution compared to flang-classic.

relevant ifort flags: /names:lowercase /assume:underscore /assume:protect_parens /fp:strict /Qopenmp-
dbg: /Od /Ob0 rel:/O2 /Ot
flang-classic: default cmake flags except for -Mrecursive -nomp.
vcpkg PR: microsoft/vcpkg#24622

So whats up with the failing tests? Are they buggy? I only see one numerical erros so I assume either real windows compiler is fine?

the following diff was necessary to build the tests with ifort.

diff --git a/TESTING/EIG/derred.f b/TESTING/EIG/derred.f
index 6df5178..997bf48 100644
--- a/TESTING/EIG/derred.f
+++ b/TESTING/EIG/derred.f
@@ -99,7 +99,7 @@
 *     ..
 *     .. External Subroutines ..
       EXTERNAL           CHKXER, DGEES, DGEESX, DGEEV, DGEEVX, DGEJSV,
-     $                   DGESDD, DGESVD, DGESVDX, DGESVQ
+     $                   DGESDD, DGESVD, DGESVDX
 *     ..
 *     .. External Functions ..
       LOGICAL            DSLECT, LSAMEN
diff --git a/TESTING/EIG/zerred.f b/TESTING/EIG/zerred.f
index d1219c0..4c21435 100644
--- a/TESTING/EIG/zerred.f
+++ b/TESTING/EIG/zerred.f
@@ -100,7 +100,7 @@
 *     ..
 *     .. External Subroutines ..
       EXTERNAL           CHKXER, ZGEES, ZGEESX, ZGEEV, ZGEEVX, ZGESVJ,
-     $                   ZGESDD, ZGESVD, ZGESVDX, ZGESVQ
+     $                   ZGESDD, ZGESVD, ZGESVDX
 *     ..
 *     .. External Functions ..
       LOGICAL            LSAMEN, ZSLECT

[martin-frbg]

You'll need to look at (and/or post the) TESTING/testing-results.txt for the significance of the failed and skipped tests - the "other" errors are usually from the error-exit tests, you will see something like "XERBLA was called from FOO with INFO=1 instead of 3" or even "XERBLA was called from FOO instead of BAR" when an unexpected error exit was taken or the error was raised at an an unexpected point in the call graph.

[Neumann-A]

For gfortran:
testing_results-static-dbg.txt
testing_results-static-rel.txt
testing_results-dbg.txt
testing_results-rel.txt

For flang-classic:
Some errors like: (also seen using ifort in debug)

testing_results-dbg.txt

For the ifort build:
testing_results-dbg.txt
testing_results-rel.txt

ifort and flang-classic same error:
Failure in STFSM, CFORM='T', SIDE='L', UPLO='U', TRANS='T', DIAG='N', M= 50, N = 50, test= 40.828

[martin-frbg]

STFSM failure is now #679 (with a more detailed discussion in the OpenBLAS issue linked there), I'd like to suggest at least raising the threshold to the "well thought out" value of 42 to remove spurious alerts.
The (approximately 200 instances of) used-before-initialized variables are probably a holdover from some very old fortran compiler that zero-initialized by default. They do not show up in the gfortran build as the default flags do not include either -Wuinitialized or -Wall, let alone -Werr which would be the closest equivalent of the MSCRT runtime abort. As a workaround, -finit-local-zero could be used with gfortran, and I believe /Qinit:zero with ifort, but I am not aware of comparable functionality in flang-classic