Online distributed graph partitioning using PT-Scotch

[abishekg7]

This PR enables the use of MPAS atmosphere core without requiring any graph decomposition files specified as input, if the atmosphere core has been built with the Scotch graph partitioning library (built separately).

Building MPAS with Scotch

Instructions to build Scotch are provided later in the description. Once Scotch has been installed, the MPAS atmosphere core can be leveraged to use online graph partitioning by setting the path to Scotch prior to building MPAS.

export SCOTCH=/path/to/scotch/installation
make nvhpc CORE=atmosphere

If MPAS is built/linked with Scotch successfully, you should see a message stating that at the end of the build output.

Usage and run-time behavior

After building MPAS with Scotch, you can still choose whether or not to use online graph partitioning by setting appropriate values for the config_block_decomp_file_prefix namelist option.

• If config_block_decomp_file_prefix+mpi_tasks points to a valid graph decomp file that already exists in the run directory, then it is used to proceed with the model run without invoking Scotch.

• If config_block_decomp_file_prefix=='' or config_block_decomp_file_prefix+mpi_tasks does not match any valid graph decomp file in the run directory, then Scotch graph partitioning is invoked. During this process, the generated partition is saved as a graph decomp file to the run directory so that it may be reused in the following runs without needing to invoke Scotch again.

If MPAS has not been built with scotch, any code paths relating to Scotch will not be taken. And an incorrect specification of config_block_decomp_file_prefix+mpi_tasks should lead to the model halting.

Downloading and building Scotch

Scotch may be obtained from its gitlab repository or as tarballs from this link

git clone https://gitlab.inria.fr/scotch/scotch.git && cd scotch 
git checkout v7.0.7

Build instructions are on this page

It can be as simple as

mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=prefix/scotch/install -DMPI_HOME=/your/mpi/path  ..
make -j5
make install

To be able to use the distributed graph partitioning provided by PT-Scotch, you will need to pass an appropriate MPI installation path during the Scotch build.

Other system pre-requisites are Bison and Flex. I have observed that an older Flex version (v2.6.1) on Eris caused the Scotch build to fail. After pulling in a more recent version of Flex v2.6.4 it seemed to build fine. Scotch documentation also references the requirement for a Bison version > 3.4.