Test segmentation fault with version 0.3.27 on RISC-V

[bedroge]

When trying to install OpenBLAS 0.3.27 with EasyBuild on a Starfive VisionFive 2 RISC-V development board, I'm getting:

 Real BLAS Test Program Results


 Test of subprogram number  1             SDOT 
                                    ----- PASS -----

 Test of subprogram number  2            SAXPY 
                                    ----- PASS -----

 Test of subprogram number  3            SROTG 
                                    ----- PASS -----

 Test of subprogram number  4             SROT 
                                    ----- PASS -----

 Test of subprogram number  5            SCOPY 
                                    ----- PASS -----

 Test of subprogram number  6            SSWAP 
                                    ----- PASS -----

 Test of subprogram number  7            SNRM2 
                                    ----- PASS -----

 Test of subprogram number  8            SASUM 
                                    ----- PASS -----

 Test of subprogram number  9            SSCAL 
                                    ----- PASS -----

 Test of subprogram number 10            ISAMAX
                                    ----- PASS -----

 Test of subprogram number 11            SROTMG
                                    ----- PASS -----

 Test of subprogram number 12            SROTM 
                                    ----- PASS -----

 Test of subprogram number 13            SDSDOT
                                    ----- PASS -----

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:

Program received signal SIGTRAP: Trace/breakpoint trap.

Backtrace for this error:
make[1]: *** [Makefile:48: level1] Error 127
make[1]: Leaving directory '/nvme/eb/build/OpenBLAS/0.3.27/GCC-13.3.0/OpenBLAS-0.3.27/test'
make: *** [Makefile:167: tests] Error 2

cc @SebastianAchilles who encountered the same error on a SiFive HiFive Unmatched board.

[martin-frbg]

Strange as the cpu on the Visionfive2 does not provide rvv support (as far as I know), so the build should be using the "generic" plain C kernels. Which compiler and build options did you use ?

[bedroge]

It's using GCC 13.3.0 with -O2 -ftree-vectorize -march=rv64gc -mabi=lp64d -fno-math-errno.

[martin-frbg]

Not reproducible with GCC 13.2.0 and default build options (which include march=rv64imafdc -mabi=lp64d)

[bedroge]

I've just started a build with EasyBuild and GCC 13.2.0 as well, let's see if that does work.

[martin-frbg]

I'm guessing it may be either the -ftree-vectorize or the way you are passing the build options... I trust that you are using identical versions of gcc and gfortran

[bedroge]

Using GCC 13.2.0 instead of 13.3.0 didn't make a difference, so I'll try changing those flags (the same ones did work fine with version 0.3.24).

[martin-frbg]

I also did not get a segfault with all the flags that you mentioned (including the -ftree-vectorize) added to the COMMON_OPT line in Makefile.rule) - the only caveat is that I am currently testing with the develop branch instead of 0.3.27, but I do not recall any change that would make a difference here. Checking out 0.3.27 now for a repeat of the test - may take some time as I'm using the shared hardware of the GCC Compile Farm

[martin-frbg]

Still not reproducible with 0.3.27 - I think there must be something else peculiar to your build or hardware. (Note that SDSDOT is actually the last test run in sblat1 - is it this or one of the other tests in the test folder that appears to be failing for you ?)

[bedroge]

I've now tried it without EasyBuild, and instead just used the GCC of my OS:

user@starfive:/nvme/openblas/OpenBLAS-0.3.27$ gcc -v | grep version
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/riscv64-linux-gnu/12/lto-wrapper
Target: riscv64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 12.2.0-10' --with-bugurl=file:///usr/share/doc/gcc-12/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-12 --program-prefix=riscv64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --disable-multilib --with-arch=rv64gc --with-abi=lp64d --enable-checking=release --build=riscv64-linux-gnu --host=riscv64-linux-gnu --target=riscv64-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (Debian 12.2.0-10) 

user@starfive:/nvme/openblas/OpenBLAS-0.3.27$ gfortran -v 
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/lib/gcc/riscv64-linux-gnu/12/lto-wrapper
Target: riscv64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 12.2.0-10' --with-bugurl=file:///usr/share/doc/gcc-12/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-12 --program-prefix=riscv64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --disable-multilib --with-arch=rv64gc --with-abi=lp64d --enable-checking=release --build=riscv64-linux-gnu --host=riscv64-linux-gnu --target=riscv64-linux-gnu
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (Debian 12.2.0-10) 

And I compiled OpenBLAS using similar commands as EasyBuild would normally use:

make -j 4 shared  BINARY='64'  CC='gcc'  FC='gfortran'  MAKE_NB_JOBS='-1'  USE_OPENMP='1'  USE_THREAD='1'  CFLAGS='-O2'
make tests  BINARY='64'  CC='gcc'  FC='gfortran'  MAKE_NB_JOBS='-1'  USE_OPENMP='1'  USE_THREAD='1

The latter fails with:

make[1]: Entering directory '/nvme/openblas/OpenBLAS-0.3.27/test'
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize -c sblat1.f  -o sblat1.o
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp  -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize  -o sblat1 sblat1.o ../libopenblas_riscv64_genericp-r0.3.27.a -lm -lpthread -lgfortran -lm -lpthread -lgfortran -L/usr/lib/gcc/riscv64-linux-gnu/12 -L/lib/riscv64-linux-gnu -L/usr/lib/riscv64-linux-gnu  -lpthread -lc -latomic 
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgfortran.a(fpu.o): in function `.L0 ':
(.text._gfortrani_set_fpu_trap_exceptions+0x82): warning: fedisableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_set_fpu_trap_exceptions+0x70): warning: feenableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_get_fpu_trap_exceptions+0x4): warning: fegetexcept is not implemented and will always fail
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgomp.a(oacc-profiling.o): in function `.L0 ':
(.text+0x7da): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize -c dblat1.f  -o dblat1.o
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp  -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize  -o dblat1 dblat1.o ../libopenblas_riscv64_genericp-r0.3.27.a -lm -lpthread -lgfortran -lm -lpthread -lgfortran -L/usr/lib/gcc/riscv64-linux-gnu/12 -L/lib/riscv64-linux-gnu -L/usr/lib/riscv64-linux-gnu  -lpthread -lc -latomic 
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgfortran.a(fpu.o): in function `.L0 ':
(.text._gfortrani_set_fpu_trap_exceptions+0x82): warning: fedisableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_set_fpu_trap_exceptions+0x70): warning: feenableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_get_fpu_trap_exceptions+0x4): warning: fegetexcept is not implemented and will always fail
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgomp.a(oacc-profiling.o): in function `.L0 ':
(.text+0x7da): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize -c cblat1.f  -o cblat1.o
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp  -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize  -o cblat1 cblat1.o ../libopenblas_riscv64_genericp-r0.3.27.a -lm -lpthread -lgfortran -lm -lpthread -lgfortran -L/usr/lib/gcc/riscv64-linux-gnu/12 -L/lib/riscv64-linux-gnu -L/usr/lib/riscv64-linux-gnu  -lpthread -lc -latomic 
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgfortran.a(fpu.o): in function `.L0 ':
(.text._gfortrani_set_fpu_trap_exceptions+0x82): warning: fedisableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_set_fpu_trap_exceptions+0x70): warning: feenableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_get_fpu_trap_exceptions+0x4): warning: fegetexcept is not implemented and will always fail
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgomp.a(oacc-profiling.o): in function `.L0 ':
(.text+0x7da): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize -c zblat1.f  -o zblat1.o
gfortran -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp  -march=rv64imafdc -mabi=lp64d -static -fno-tree-vectorize  -o zblat1 zblat1.o ../libopenblas_riscv64_genericp-r0.3.27.a -lm -lpthread -lgfortran -lm -lpthread -lgfortran -L/usr/lib/gcc/riscv64-linux-gnu/12 -L/lib/riscv64-linux-gnu -L/usr/lib/riscv64-linux-gnu  -lpthread -lc -latomic 
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgfortran.a(fpu.o): in function `.L0 ':
(.text._gfortrani_set_fpu_trap_exceptions+0x82): warning: fedisableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_set_fpu_trap_exceptions+0x70): warning: feenableexcept is not implemented and will always fail
/usr/bin/ld: (.text._gfortrani_get_fpu_trap_exceptions+0x4): warning: fegetexcept is not implemented and will always fail
/usr/bin/ld: /usr/lib/gcc/riscv64-linux-gnu/12/libgomp.a(oacc-profiling.o): in function `.L0 ':
(.text+0x7da): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat1
 Real BLAS Test Program Results


 Test of subprogram number  1             SDOT 
                                    ----- PASS -----

 Test of subprogram number  2            SAXPY 
                                    ----- PASS -----

 Test of subprogram number  3            SROTG 
                                    ----- PASS -----

 Test of subprogram number  4             SROT 
                                    ----- PASS -----

 Test of subprogram number  5            SCOPY 
                                    ----- PASS -----

 Test of subprogram number  6            SSWAP 
                                    ----- PASS -----

 Test of subprogram number  7            SNRM2 
                                    ----- PASS -----

 Test of subprogram number  8            SASUM 
                                    ----- PASS -----

 Test of subprogram number  9            SSCAL 
                                    ----- PASS -----

 Test of subprogram number 10            ISAMAX
                                    ----- PASS -----

 Test of subprogram number 11            SROTMG
                                    ----- PASS -----

 Test of subprogram number 12            SROTM 
                                    ----- PASS -----

 Test of subprogram number 13            SDSDOT
                                    ----- PASS -----

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
Segmentation fault
make[1]: *** [Makefile:48: level1] Error 139
make[1]: Leaving directory '/nvme/openblas/OpenBLAS-0.3.27/test'

[bedroge]

I've just tried the exact same thing for version 0.3.26, and that does work fine.

[martin-frbg]

Rerunning my build in the gcc compile farm now - maybe it is the USE_OPENMP=1 that makes the difference, I did not notice that in your initial report. (There have been some OpenMP-related changes in 0.3.27, but on the other hand they should be affecting all platforms)

[bedroge]

It looks like that's indeed causing the failures. I've tried the same thing now with 0.3.27 and USE_OPENMP=0, and then everything works fine.

[martin-frbg]

Reproduced, but gdb's backtraces only lead to the implementation of the CLOSE() function in the Fortran runtime library. (I guess this could still hint at a memory management problem, but sadly that gccfarm machine does not appear to have valgrind installed). The C-only openblas_utest and openblas_utest_ext run without errors.

[martin-frbg]

Bisecting now, but this will probably take me until tomorrow due to real life.

[martin-frbg]

Bisected to

bef47917bd72f35c151038fee0cf485445476863 is the first bad commit
commit bef47917bd72f35c151038fee0cf485445476863
Author: Heller Zheng <heller.zheng@microchip.com>
Date:   Tue Nov 15 00:06:25 2022 -0800

    Initial version for riscv sifive x280

not sure yet if that is actually true (RISCV64_GENERIC does not use any of the *_rvv.c kernel files added by that PR, but compiler flags for TARGET=RISCV64_GENERIC were introduced at the same time as well). Also this used to live on the risc-v branch for more than a year, so I imagine it could just be the merge in general that went wrong.