HOME

Data-Intensive Applications on HPC Using Hadoop, Spark and RADICAL-Cybertools (Supercomputing 2015 Tutorial)

Thank you for the great tutorial. We had a total of 63 participants! 31 participants successfully participated in the exercises running Hadoop, Spark and RADICAL-Pilot. See page below for archived materials!

Material

• Slides
• [Notebooks] (http://nbviewer.ipython.org/github/radical-cybertools/supercomputing2015-tutorial/tree/master/)

Links to Tool Documentations

• RADICAL-Pilot
• Hadoop
• Spark

Infrastructure

This tutorial is based on a set of iPython notebooks, which allows us to interactively create and execute code from the web browser. If you are not familiar with iPython notebooks, we recommend to take some time to familiarize yourself with iPython:

• Documentation: http://ipython.readthedocs.org/en/stable/overview.html

The archived notebooks can be viewed with [NBViewer] (http://nbviewer.ipython.org/github/radical-cybertools/supercomputing2015-tutorial/tree/master/).

Exercises (Environment Not Active After Tutorial)

Please walk through the Examples located in our Jupyter notebooks:

http://sc15-aws.radical-cybertools.org

1. Log in using your assigned tutorial user (it is the same user as your Linux/SSH user):

   

   You will see an empty list folder in the browser:

   

2. Checkout the Tutorials Github repository by creating an empty notebook (Python2). You can execute bash commands using the ! before the command. Create a cell with the following content and execute it by pressing shift-enter:

    !git clone https://github.com/radical-cybertools/supercomputing2015-tutorial.git
   

   

   You will see a new folder supercomputing2015-tutorial:

   

3. Go back to the overview screen and enter the supercomputing2015-tutorial folder:

   

   The tutorial overview notebooks gives an overview and links to the individual notebooks:

   

Please go to the Jupyterhub login page and execute step 1-3.

Troubleshooting

1. Log into edge node (also referred to as master node) with your tutorial user:

    ssh <user>@sc15-aws.radical-cybertools.org
   

2. Test keyless login on Tutorial Edge Node:

    ssh localhost /bin/date
   

   This should work out of the box with the SSH key that has been automatically setup with your account. If it fails, you need to create a keyless login:

    ssh-keygen -t rsa
    cd .ssh
    cat id_rsa.pub >> authorized_keys
    chmod 700 ~/.ssh
    chmod 600 ~/.ssh/authorized_keys
   

3. Verify that there is a notebooks directory ~/notebooks in your home folder!