HDF5 parallel test suite on Lustre hangs with OMPI 4.0.5

[bodgerer]

• Open MPI: 4.0.1
• Operating system/version: centos7.6
• Computer hardware: Intel Skylake
• Network type: Mellanox EDR InfiniBand (although not used in steps to reproduce)

I'm opening this ticket because HDF5 1.8.21 hangs on its parallel test suite with OpenMPI 4.0.1 when run on our Lustre 2.12.2 parallel filesystem. They run to completion when run on an ext4 filesystem.

The following script reproduces both the openmpi/hdf5 build and the issue with the test suite. Can someone help, please?

Thanks,

Mark

#!/bin/bash

# We have run this on a centos7.6 system with an idle Lustre 2.12.2
# filesystem mounted. Same Lustre version on both client and servers.
#
# If we run the script from a location on the Lustre filesystem,
# the hdf5 test "testphdf5" hangs until it is terminated by its 20
# minute alarm. Increasing the alarm timout makes no difference
# (it's a new, idle filesystem). The last lines that the test
# printed was 6 copies of this:
#
# Testing  -- multi-chunk collective chunk io (cchunk3)
#
# If we run the script from a location on an ext4 filesystem,
# the hdf5 test "testphdf5" completes (although fails hdf5 1.8.21's
# t_pflush1 test, which I believe is a known, separate issue)

set -x
set -e

# (needed on our system to ensure we are using the OS-provided
# version of GCC, etc.)
module purge || true

test -d build || mkdir build
test -d src || mkdir src

prefix=`pwd`/build
export PATH=${prefix}/bin:$PATH

cd src

# openmpi

wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.bz2 
tar xf openmpi-4.0.1.tar.bz2
cd openmpi-4.0.1

# (we get the same test behaviour without "--with-io-romio-flags"; however,
# OpenMPI's ROMIO on Lustre is clearly broken without it - trying to use an
# MPI-IO hint to stripe a file didn't work)

./configure --prefix=$prefix \
   --with-io-romio-flags=--with-file-system=lustre+ufs \
   --enable-mpi1-compatibility

make -j12
make install
cd ..

# (ignore infiniband for now if we have it - hdf5 tests only need one host.
# This avoids some warning messages we can ignore)
export OMPI_MCA_btl=^openib

# (disable openmpi's own MPI-IO implementation - recommended by the hdf5
# folks. I believe openmpi defaults to ROMIO on Lustre anyway)
export OMPI_MCA_io=^mpio


# hdf5

wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.21/src/hdf5-1.8.21.tar.bz2 
tar xf hdf5-1.8.21.tar.bz2
cd hdf5-1.8.21

export CC=mpicc
export CXX=mpicxx
export FC=mpif90
export F77=mpif77
export F90=mpif90

./configure --prefix=$prefix --enable-parallel
make -j12
make check
make install
cd ..

[edgargabriel]

@hppritcha not entirely sure what to do about this one, since it is using romio, not ompio.

[bodgerer]

Have retested with HDF5 1.10.7 and openmpi 4.0.5 and it is still an issue.

Digging into it a little bit more, it hangs on testphdf5, but only during the cchunk3 or tldsc tests.

Is no-one maintaining openmpi's MPI-IO on Lustre?

[rhc54]

I believe the answer is "yes", but not necessarily the ROMIO distribution. I don't know why the HDF5 folks recommend using ROMIO over ompio - have you tried it? You are more likely to get help from that direction.

[bodgerer]

Hi Ralph,

The old comment of mine in the script is misleading, sorry - it's not me nor the hdf5 project currently choosing to use romio: openmpi defaults to romio on lustre and the last time it was mentioned in the openmpi release notes (2.0.0), it was recommended. A few days ago, Edgar told me on the user list that there were performance reasons for setting romio on lustre by default.

So we are in the situation where openmpi 4 on lustre does not pass hdf5's parallel tests using its own defaults and recommendation.

But yes, openmpi 4.0.5 + ucx 1.9.0 + hdf5 1.10.7 passes hdf5's tests if ompio is selected with the following provisos:

• test t_bigio in particular is unusually slow (40 minutes on my system) and so needs HDF5_ALARM_SECONDS=3600 to avoid being killed prematurely. I guess this is related to the performance issue(s) Edgar mentioned.
• UCX_LOG_LEVEL=ERROR should be set, otherwise I've seen many warning messages emitted, which the hdf5 test scripts interpret as an error:

Best,

Mark

[bodgerer]

I no longer have access to the original system, but ROMIO on a power9 system seeing the same problem on openmpi 4.0.5 is fixed by this pull request:

#3975

See:

https://www.mail-archive.com/users@lists.open-mpi.org//msg34187.html

[markalle]

I've been putting my romio fixes into mpich here:
pmodels/mpich#4943 [accepted]
pmodels/mpich#4948
pmodels/mpich#4956
and I've made an OMPI branch off of v4.0.x with all the above pushed into it. I can post it now or wait till the above all get accepted into mpich first.

I think just the first change above is the one that matters to HDF5. Anyway with the above added to v4.0.x my hdf5 run passes following the above instructions (using hdf5-1.10.7.tar.bz2 on PPC with gpfs and either romio321 or ompio).

[ggouaillardet]

@markalle thanks a lot!

Please open two PR for the v4.0.x and v4.1.x branches.

The way to move forward is imho to update ROMIO (see #8279), but I do not see that happening in the current release branches, so I am afraid patching them is our best option for the time being.