test allocate_as_barrier_proc hangs

[mexas]

FreeBSD laptop, 2 cores, opencoarrays 1.8.4 with GCC 7.0.0,
on make test, hangs at:

# make test
===>  Testing for opencoarrays-1.8.4
(cd /usr/ports/lang/opencoarrays/work/.build && ctest)
Test project /usr/ports/lang/opencoarrays/work/.build
      Start  1: initialize_mpi
 1/40 Test  #1: initialize_mpi ...................   Passed    0.06 sec
      Start  2: register
 2/40 Test  #2: register .........................   Passed    0.07 sec
      Start  3: register_vector
 3/40 Test  #3: register_vector ..................***Failed  Required regular expression not found.Regex=[Test passed.
]  5.18 sec
      Start  4: register_alloc_vector
 4/40 Test  #4: register_alloc_vector ............***Failed  Required regular expression not found.Regex=[Test passed.
]  5.04 sec
      Start  5: allocate_as_barrier
 5/40 Test  #5: allocate_as_barrier ..............***Failed  Required regular expression not found.Regex=[Test passed.
]  5.68 sec
      Start  6: allocate_as_barrier_proc

top shows

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME     CPU COMMAND
 2004 root        21    0   232M   202M bo_wwa  1   0:03   6.18% allocate_as_barrier
 2021 root        20    0   232M 17196K range   0   0:04   0.00% allocate_as_barrier
 2022 root        20    0   232M 17108K range   0   0:04   0.00% allocate_as_barrier
 2020 root        20    0   232M 17204K range   0   0:04   0.00% allocate_as_barrier
 2019 root        20    0   232M 17204K range   0   0:03   0.00% allocate_as_barrier
 2018 root        20    0   232M 17260K range   0   0:03   0.00% allocate_as_barrier
 2016 root        20    0   232M 17332K range   1   0:03   0.00% allocate_as_barrier
 2012 root        20    0   232M 17152K range   0   0:03   0.00% allocate_as_barrier
 2010 root        20    0   232M 17092K range   0   0:03   0.00% allocate_as_barrier
 2014 root        20    0   232M 17224K range   0   0:03   0.00% allocate_as_barrier
 2005 root        20    0   232M 17056K range   0   0:03   0.00% allocate_as_barrier
 1997 root        20    0   232M 17264K range   0   0:01   0.00% allocate_as_barrier

I put here only allocate_as_barrier threads, excluding all other processes.

Killing the process with CTRL/C results in a system panic.
I'll report this to FreeBSD too.

I haven't seen this on my other laptop, which has identical FreeBSD,
opencoarrays and GCC versions. The only difference is that the other
laptop has 4 cores.

Thanks

Anton

[vehre]

Hi Anton,

I doubt this has something to do with allocate as barrier when previous tests are failing already. What's the output of the first test failing when you run it manually?

Andre

[mexas]

# mpiexec -np 2 src/tests/unit/init_register/register_vector 

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:
#0  0x801eb678e in ???
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401e20 in ???
#4  0x4019b9 in ???
#5  0x401b4e in ???
#6  0x80062afff in ???
#0  0x801eb678e in ???
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401e20 in ???
#4  0x4019b9 in ???
#5  0x401b4e in ???
#6  0x80062afff in ???

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

I guess I have to rebuild with -g?

[mexas]

Here you go:

# mpirun -np 2 src/tests/unit/init_register/register_vector 

Program received signal SIGILL: Illegal instruction.

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:

Backtrace for this error:
#0  0x801eb678e in ???
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401e20 in register2
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_vector.f90:34
#4  0x4019b9 in main
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_vector.f90:43
#0  0x801eb678e in ???
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401e20 in register2
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_vector.f90:34
#4  0x4019b9 in main
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_vector.f90:43

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

[mexas]

Probably means I need to rebuild mpich with -g too.

[vehre]

That is strange. There is no error in the mpi_caf reported, which we are looking for. Did you rebuild from a clean tree? So that no left overs were present.

[mexas]

yes, clean dir prior to build from

sourceryinstitute-OpenCoarrays-1.8.4_GH0.tar.gz

Different lines in other tests:

]# mpirun -np 2 src/tests/unit/init_register/register_alloc_vector 

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:
#0  0x801eb678e in ???
#0  0x801eb678e in ???
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401d48 in register3
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_alloc_vector.f90:35
#4  0x4019c9 in main
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_alloc_vector.f90:49
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401d48 in register3
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_alloc_vector.f90:35
#4  0x4019c9 in main
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/register_alloc_vector.f90:49

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

or

# mpirun -np 2 src/tests/unit/init_register/allocate_as_barrier   

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:

Program received signal SIGILL: Illegal instruction.

Backtrace for this error:
#0  0x801eb678e in ???
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401e01 in alloc_as_barrier
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/allocate_as_barrier.f90:13
#4  0x401a79 in main
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/allocate_as_barrier.f90:24
#0  0x801eb678e in ???
#1  0x801eb5d6e in ???
#2  0x7ffffffff192 in ???
#3  0x401e01 in alloc_as_barrier
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/allocate_as_barrier.f90:13
#4  0x401a79 in main
        at /usr/ports/lang/opencoarrays/work/OpenCoarrays-1.8.4/src/tests/unit/init_register/allocate_as_barrier.f90:24

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

[zbeekman]

@mexas is this still present in 1.8.10? If not, let's close this issue

[mexas]

I'm now testing 1.8.12 on linux 2.6.32-642.6.2.el6.x86_64 with GCC 7.1.0,
mvapich2-2.3a.

I get:

ctest
...
The following tests FAILED:
          6 - allocate_as_barrier_proc (Failed)
         21 - syncall (Failed)
         22 - syncimages (Failed)
         23 - syncimages2 (Failed)
         27 - syncimages_status (Failed)
         30 - simpleatomics (Failed)
Errors while running CTest

However, these tests seem to pass when run manually:

[mexas@newblue1 build]$ which mpiexec
/cm/shared/apps/mvapich2-2.3a/bin/mpiexec
[mexas@newblue1 build]$ mpiexec --version
HYDRA build details:
    Version:                                 3.2
    Release Date:                            Wed Mar 29 16:05:27 EDT 2017
    CC:                              /cm/shared/languages/GCC-7.1.0/bin/gcc    
    CXX:                             /cm/shared/languages/GCC-7.1.0/bin/g++    
    F77:                             /cm/shared/languages/GCC-7.1.0/bin/gfortran   
    F90:                             /cm/shared/languages/GCC-7.1.0/bin/gfortran   
    Configure options:                       '--disable-option-checking' '--prefix=/cm/shared/apps/mvapich2-2.3a' '--with-ib-include=/usr/include' '--with-ib-libpath=/usr/lib64' '--with-psm-include=/usr/include' '--with-psm-lib=/usr/lib64' 'CC=/cm/shared/languages/GCC-7.1.0/bin/gcc' 'CXX=/cm/shared/languages/GCC-7.1.0/bin/g++' 'FC=/cm/shared/languages/GCC-7.1.0/bin/gfortran' 'F77=/cm/shared/languages/GCC-7.1.0/bin/gfortran' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -DNDEBUG -DNVALGRIND -O2' 'LDFLAGS=-L/usr/lib64 -L/lib -L/lib -L/lib -Wl,-rpath,/lib -L/lib -Wl,-rpath,/lib -L/usr/lib64 -L/lib -L/lib' 'LIBS=-libmad -lrdmacm -libumad -libverbs -ldl -lrt -lm -lpthread ' 'CPPFLAGS=-I/usr/include -I/panfs/panasas01/isys/iszcjw/Software/Anton/mvapich2-2.3a/src/mpl/include -I/panfs/panasas01/isys/iszcjw/Software/Anton/mvapich2-2.3a/src/mpl/include -I/panfs/panasas01/isys/iszcjw/Software/Anton/mvapich2-2.3a/src/openpa/src -I/panfs/panasas01/isys/iszcjw/Software/Anton/mvapich2-2.3a/src/openpa/src -D_REENTRANT -I/panfs/panasas01/isys/iszcjw/Software/Anton/mvapich2-2.3a/src/mpi/romio/include -I/include -I/include -I/usr/include -I/include -I/include'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:       
    Demux engines available:                 poll select
...
[mexas@newblue1 build]$ mpiexec -np 4 ./src/tests/unit/init_register/allocate_as_barrier_proc 
 Test passed.
[mexas@newblue1 build]$ mpiexec -np 8 ./src/tests/unit/init_register/allocate_as_barrier_proc 
 Test passed.
[mexas@newblue1 build]$ mpiexec -np 16 ./src/tests/unit/init_register/allocate_as_barrier_proc 
 Test passed.
[mexas@newblue1 build]$ mpiexec -np 1 ./src/tests/unit/init_register/allocate_as_barrier_proc 
[newblue1:mpi_rank_0][rdma_find_network_type] QLogic IB card detected in system
[newblue1:mpi_rank_0][rdma_find_network_type] Please re-configure the library with the '--with-device=ch3:psm' configure option for best performance
[mexas@newblue1 build]$ mpiexec -np 2 ./src/tests/unit/init_register/allocate_as_barrier_proc 
 Test passed.
...
[mexas@newblue1 build]$ mpiexec -np 2 ./src/tests/unit/sync/syncall                           
 Test passed.
[mexas@newblue1 build]$ mpiexec -np 16 ./src/tests/unit/sync/syncall 
 Test passed.
...
[mexas@newblue1 build]$ mpiexec -np 2 ./src/tests/unit/sync/syncimages
 Test passed.
[mexas@newblue1 build]$ mpiexec -np 7 ./src/tests/unit/sync/syncimages
 Test passed.
[mexas@newblue1 build]$ mpiexec -np 13 ./src/tests/unit/sync/syncimages
 Test passed.
...
[mexas@newblue1 build]$ mpiexec -np 13 ./src/tests/unit/sync/syncimages2
 Test passed.
[mexas@newblue1 build]$ mpiexec -np 3 ./src/tests/unit/sync/syncimages2
 Test passed.

The last test, simpleatomics, I cannot even find:

[mexas@newblue1 build]$ pwd
/panfs/panasas01/mech/mexas/soft/OpenCoarrays-1.8.12/build
[mexas@newblue1 build]$ find . -name "simplea*"
[mexas@newblue1 build]$

Perhaps it wasn't built?

[mexas]

probably too old now - both GCC and opencoarrays have moved on, closing

[zbeekman]

OK... FYI I've greatly simplified and updated the issue templates... hopefully it will be concise enough for you to use, and it should help reduce confusion about the details of the bugs/systems/software stacks involved.