Merge branch 'l1ll1-mpi_update' into docs/2.3

Author: vsoch

pages/docs/admin-docs/docs-hpc.md

@@ -10,28 +10,67 @@ One of the architecturally defined features in Singularity is that it can execut
 
 Additionally, because Singularity is not emulating a full hardware level virtualization paradigm, there is no need to separate out any sandboxed networks or file systems because there is no concept of user-escalation within a container. Users can run Singularity containers just as they run any other program on the HPC resource.
 
-### Workflows
+## Workflows
 We are in the process of developing Singularity Hub, which will allow for generation of workflows using Singularity containers in an online interface, and easy deployment on standard research clusters (e.g., SLURM, SGE). Currently, the Singularity core software is installed on the following research clusters, meaning you can run Singularity containers as part of your jobs:
 
 - The <a href="http://sherlock.stanford.edu" target="_blank" class="no-after">Sherlock cluster</a> at <a href="https://srcc.stanford.edu/" class="no-after" target="_blank">Stanford University</a>
 - <a href="https://www.xsede.org/news/-/news/item/7624" target="_blank" class="no-after">SDSC Comet and Gordon</a> (XSEDE)
 
 ### Integration with MPI
-Another result of the Singularity architecture is the ability to properly integrate with the Message Passing Interface (MPI). Work has already been done for out of the box compatibility with Open MPI (both in Open MPI v2.x as well as part of Singularity). Here is a document that illustrates the Open MPI/Singularity workflow:
-
-From the above image you can follow the invocation pathway:
-1. mpirun is called by the resource manager or the user directly from a shell
-2. Open MPI then calls the process management daemon (ORTED)
-3. The ORTED process launches the Singularity container requested by the mpirun command
-4. Singularity builds the container and namespace environment
-5. Singularity then launches the MPI application within the container
-6. The MPI application launches and loads the Open MPI libraries
-7. The Open MPI libraries connect back to the ORTED process via the Process Management Interface (PMI)
-8. At this point the processes within the container run as they would normally directly on the host at full bandwidth! This entire process happens behind the scenes, and from the user's perspective running via MPI is as simple as just calling mpirun on the host as they would normally.
-
-#### Code
-Below is an example snippet of building and installing OpenMPI into a container and then running an example MPI program through Singularity. 
- 
+Another result of the Singularity architecture is the ability to properly integrate with the Message Passing Interface (MPI). Work has already been done for out of the box compatibility with Open MPI (both in Open MPI v2.x as well as part of Singularity). The Open MPI/Singularity workflow works as follows:
+
+ 1. mpirun is called by the resource manager or the user directly from a shell
+ 2. Open MPI then calls the process management daemon (ORTED)
+ 3. The ORTED process launches the Singularity container requested by the mpirun command
+ 4. Singularity builds the container and namespace environment
+ 5. Singularity then launches the MPI application within the container
+ 6. The MPI application launches and loads the Open MPI libraries
+ 7. The Open MPI libraries connect back to the ORTED process via the Process Management Interface (PMI)
+ 8. At this point the processes within the container run as they would normally directly on the host at full bandwidth! This entire process happens behind the scenes, and from the user's perspective running via MPI is as simple as just calling mpirun on the host as they would normally.
+
+Below are example snippets of building and installing OpenMPI into a container and then running an example MPI program through Singularity.
+
+
+#### MPI Development Example
+
+**What are supported Open MPI Version(s)?** 
+To achieve proper container'ized Open MPI support, you must use Open MPI version 2.1. Open MPI version 2.1.0 includes a bug in its configure script affecting some interfaces (at least Mellanox cards operating in RoCE mode using libmxm). For this reason, we show the example first. 
+
+```bash
+$ # Include the appropriate development tools into the container (notice we are calling
+$ # singularity as root and the container is writeable)
+$ sudo singularity exec -w /tmp/Centos-7.img yum groupinstall "Development Tools"
+$
+$ # Obtain the development version of Open MPI
+$ wget https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.0.tar.bz2
+$ tar jtf openmpi-2.1.0.tar.bz2
+$ cd openmpi-2.1.0
+$
+$ # Build OpenMPI in the working directory, using the tool chain within the container
+$ # This step is unusual in a stable release but there is a bug in the configure script
+$ # affecting some interfaces
+$ singularity exec /tmp/Centos-7.img ./autogen.pl
+$ singularity exec /tmp/Centos-7.img ./configure --prefix=/usr/local
+$ singularity exec /tmp/Centos-7.img make
+$
+$ # Install OpenMPI into the container (notice now running as root and container is writeable)
+$ sudo singularity exec -w -B /home /tmp/Centos-7.img make install
+$
+$ # Build the OpenMPI ring example and place the binary in this directory
+$ singularity exec /tmp/Centos-7.img mpicc examples/ring_c.c -o ring
+$
+$ # Install the MPI binary into the container at /usr/bin/ring
+$ sudo singularity copy /tmp/Centos-7.img ./ring /usr/bin/
+$
+$ # Run the MPI program within the container by calling the MPIRUN on the host
+$ mpirun -np 20 singularity exec /tmp/Centos-7.img /usr/bin/ring
+
+```
+
+#### Code Example
+
+The following example (using their master) should work fine on most hardware but if you have an issue, try running this example below:
+
 ```bash
 $ # Include the appropriate development tools into the container (notice we are calling
 $ # singularity as root and the container is writeable)
@@ -92,6 +131,3 @@ Process 17 exiting
 Process 18 exiting
 Process 19 exiting
 ```
-
-### Notes
-Supported Open MPI Version(s): To achieve proper container'ized Open MPI support, you must use Open MPI version 2.1 which at the time of this writing has not been released yet. The above example builds from the current master development branch of Open MPI.