Adding contributions for GPUS and gist support (#75)

* initial recipe for binding in some hostlibs

* integrating contribution for gpus, and adding gist template to docs/2.3

Author: vsoch

Gemfile

@@ -1,5 +1,5 @@
 source "https://rubygems.org"
 
-# gem "rails"
+#gem "rails"
 gem 'github-pages'
-gem 'jekyll'
+gem 'jekyll'

_includes/gist.html

@@ -0,0 +1,9 @@
+<style>
+/* Better styles for embedding GitHub Gists */
+.gist{font-size:13px;line-height:20px;margin-bottom:20px;width:100%}
+.gist pre{font-family:Menlo,Monaco,'Bitstream Vera Sans Mono','Courier New',monospace !important}
+.gist-meta{font-family:Helvetica,Arial,sans-serif;font-size:13px !important}
+.gist-meta a{color:#26a !important;text-decoration:none}
+.gist-meta a:hover{color:#0e4071 !important}
+</style>
+<script src="https://gist.github.com/{{ include.username }}/{{ include.id  }}.js?file={{ include.file }}"></script>

_posts/recipes/2017-05-09-hostlibs-gpus-and-mpi.md

@@ -0,0 +1,68 @@
+---
+title:  "Using Host libraries: GPU drivers and OpenMPI BTLs"
+category: recipes
+permalink: tutorial-gpu-drivers-open-mpi-mtls
+---
+
+Singularity does a fantastic job  of isolating you from the host so you don't
+have to muck about with `LD_LIBRARY_PATH`, you just get exactly the library
+versions you want. However, in some situations you need to use library
+versions that match host exactly. Two common ones are NVIDIA gpu
+driver user-space libraries, and OpenMPI transport drivers for high performance
+networking. There are many ways to solve these problems. Some people build a container and
+copy the version of the libs (installed on the host) into the container. 
+
+{% include toc.html %}
+
+## What We will learn today
+This document describes how to use a bind mount, symlinks and ldconfig so that when the host
+libraries are updated the container does not need to be rebuilt.
+
+**Note** this tutorial is tested with Singularity <a href="https://github.com/singularityware/singularity/commit/945c6ee343a1e6101e22396a90dfdb5944f442b6" target="_blank">commit 945c6ee343a1e6101e22396a90dfdb5944f442b6</a>,
+ which is part of the (current) development branch, and thus it should work with version 2.3 
+when that is released. The version of OpenMPI used is 2.1.0 (versions above 2.1 should work).
+
+## Environment
+
+In our environment we run CentOS 7 hosts with:
+
+  1. slurm located on `/opt/slurm-<version>` and the slurm user `slurm`
+  2. Mellanox network cards with drivers installed to `/opt/mellanox` (
+    Specifically we run a RoCEv1 network for Lustre and MPI communications)
+  3. NVIDIA GPUs with drivers installed to `/lib64`
+  4. OpenMPI (by default) for MPI processes
+
+## Creating your image
+Since we are building an ubuntu image, it may be easier to create an ubuntu VM
+to create the image. Alternatively you can follow the recipe
+<a href="/building-ubuntu-rhel-host" target="_blank"> here</a>.
+
+Use the following def file to create the image.
+
+{% include gist.html username='l1ll1' id='89b3f067d5b790ace6e6767be5ea2851' file='hostlibs.def' %}
+
+The mysterious `wget` line gets a list of all the libraries that the CentOS host
+has in `/lib64` that *we* think its safe to use in the container. Specifically
+these are things like nvidia drivers.
+
+{% include gist.html username='l1ll1' id='89b3f067d5b790ace6e6767be5ea2851' file='desired_hostlibs.txt' %}
+
+Also note:
+
+1.  in `hostlibs.def` we create a slurm user. Obviously if your `SlurmUser` is different you should change this name.
+2.  We make directories for `/opt` and `/usr/local/openmpi`. We're going to bindmount these from the host so we get all the bits of OpenMPI and Mellanox and Slurm that we need.
+
+
+## Executing your image
+On our system we do:
+
+```
+SINGULARITYENV_LD_LIBRARY_PATH=/usr/local/openmpi/2.1.0-gcc4/lib:/opt/munge-0.5.11/lib:/opt/slurm-16.05.4/lib:/opt/slurm-16.05.4/lib/slurm:/desired_hostlibs:/opt/mellanox/mxm/lib/
+export SINGULARITYENV_LD_LIBRARY_PATH
+```
+
+then
+
+```
+srun  singularity exec -B /usr/local/openmpi:/usr/local/openmpi -B /opt:/opt -B /lib64:/all_hostlibs hostlibs.img <path to binary>
+```

pages/docs/admin-docs/docs-hpc.md

@@ -10,6 +10,7 @@ One of the architecturally defined features in Singularity is that it can execut
 
 Additionally, because Singularity is not emulating a full hardware level virtualization paradigm, there is no need to separate out any sandboxed networks or file systems because there is no concept of user-escalation within a container. Users can run Singularity containers just as they run any other program on the HPC resource.
 
+
 ## Workflows
 We are in the process of developing Singularity Hub, which will allow for generation of workflows using Singularity containers in an online interface, and easy deployment on standard research clusters (e.g., SLURM, SGE). Currently, the Singularity core software is installed on the following research clusters, meaning you can run Singularity containers as part of your jobs:
 
@@ -30,6 +31,10 @@ Another result of the Singularity architecture is the ability to properly integr
 
 Below are example snippets of building and installing OpenMPI into a container and then running an example MPI program through Singularity.
 
+#### Tutorials
+
+ - <a href="/tutorial-gpu-drivers-open-mpi-mtls">Using Host libraries: GPU drivers and OpenMPI BTLs</a>
+
 
 #### MPI Development Example
 

pages/docs/contributing/contributing-docs.md

@@ -66,9 +66,29 @@ Jekyll has this thing they call <a href="https://jekyllrb.com/docs/frontmatter/"
 and then write your post after it in <a href="https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet" target="_blank">Markdown Syntax</a>. All and any HTML tags are fair game as well! Once you add the post, if you have set the category correctly to "news" it should show up in the site feed. It's that easy!
 
 ## Contributing to a Page
-
 All of the pages are in the <a href="https://github.com/singularityware/singularityware.github.io/blob/master/pages" target="_blank">pages</a> folder, organized in a somewhat logical manner. If you want to edit content for a page, just edit the corresponding file. If you need to do something more advanced like edit a sidebar, you should look at the <a href="https://github.com/singularityware/singularityware.github.io/blob/master/_data/sidebars" target="_blank">sidebar data</a> yml documents, which render into the navigation.
 
+
+### Adding gists
+You can embed a <a href="https://gist.github.com">Github Gist</a> directly in the page, and we encourage you to keep it under your username to maintain credit for the work. To reduce the additional gem dependencies, instead of using the official `gist include` we have our own wrapper for it that will turn this:
+ 
+```bash
+{{ "{% include gist.html username='l1ll1' id='89b3f067d5b790ace6e6767be5ea2851' file='hostlibs.def'" }}%}
+```
+
+into this:
+
+{% include gist.html username='vsoch' id='49709c8ed549155d0a15bde48e893588' file='Singularity' %}
+
+We start with a url that looks like `https://gist.github.com/vsoch/49709c8ed549155d0a15bde48e893588`, under which there is a file called `Singularity`, and the variables are mapped as follows:
+
+- username: `vsoch`
+- id: `49709c8ed549155d0a15bde48e893588`
+- file: `Singularity`
+
+You can put this single line anywhere on the site, and it will render the gist automatically.
+
+
 ### Recording demos
 
 If you think something could be better explained with a demo, record and embed one! Here's how: