Unrecognized alias: '--profile=xxx', it will probably have no effect.

[k-dahl]

The --profile option in jupyter appears to be ignored now when it's run with the notebook command. The usage for it still lists:

Examples

ipython notebook                       # start the notebook
ipython notebook --profile=sympy       # use the sympy profile
ipython notebook --certfile=mycert.pem # use SSL/TLS certificate

[minrk]

Sorry, missed that in the examples. Fixed by #310.

[k-dahl]

Out of curiosity, and to possibly clear up some confusion that I have seen on stackoverflow and such, how would one now specify startup initialization type options for Jupyter?

A specific scenario I am thinking of is with pySpark.

[Carreau]

See the ML discussion:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/jupyter/7q02jjksvFU

but you shouldn't need a profile for that. PySpark can just be a kernel, if you really want it to be.
a kernel is basically just a way of launching a process, so it can be different envs, or different location.
some national lab have name kernel depending on which physical machine the process will run for example.

[vherasme]

Hi. I am trying to create a profile for pyspark too. Could you please tell me how to proceed ? Thanks

[Carreau]

There is no notion of profile in jupyter and for the notebook.

It's roughly like asking to dual boot a computer because you want to use vim and emacs,
and get 2 hard drive just to set your $EDITOR differently.

As stated in the mailing list thread, you can if you like, it would be something like

$ JUPTYER_CONFIG_DIR=~/jupyter_pyspark_foo jupyter notebook

If should auto create the needed files in ~/jupyter_pyspark_foo, but it is likely not what you want.

You most likely just want a separate kernel, or just import pySpark as a library. Still without knowing more of what you want to do it's hard to give you an answer...

[vherasme]

I would like to use pySpark in the ipython notebook. Either by calling it
as a library or by creating a profile/kernel/ etc.

On 25 August 2015 at 11:00, Matthias Bussonnier notifications@github.com
wrote:

There is no notion of profile in jupyter and for the notebook.

It's roughly like asking to dual boot a computer because you want to use
vim and emacs,
and get 2 hard drive just to set your $EDITOR differently.

As stated in the mailing list thread, you can if you like, it would be
something like

$ JUPTYER_CONFIG_DIR=~/jupyter_pyspark_foo jupyter notebook

If should auto create the needed files in ~/jupyter_pyspark_foo, but it
is likely not what you want.

You most likely just want a separate kernel, or just import pySpark as a
library. Still without knowing more of what you want to do it's hard to
give you an answer...

—
Reply to this email directly or view it on GitHub
#309 (comment).

[Carreau]

Ok, here is what I just did durring the last 1/2 h, for me on OS X

• Install apache-spark ($ brew install apache-spark)
• install findspark ( pip install -e . after cloning https://github.com/minrk/findspark, and cd findspark)
• install java (from here)
• fire a notebook (jupyter notebook)

enter the following:

import findspark
import os
findspark.init()

import pyspark
sc = pyspark.SparkContext()
lines = sc.textFile(os.path.exapnduser('~/dev/ipython/setup.py'))
lines_nonempty = lines.filter( lambda x: len(x) > 0 )
lines_nonempty.count()

execute :

221

Yayyyyy !

[Carreau]

(Note, installing/downloading java took 20 minutes)

[vherasme]

After running:
import findspark
import os
findspark.init()

import pyspark
sc = pyspark.SparkContext()

I get this error:

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-1-0e2dcc62fef1> in <module>()
      4 
      5 import pyspark
----> 6 sc = pyspark.SparkContext()

/Users/victor/Downloads/spark-1.4.1/python/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    108         """
    109         self._callsite = first_spark_call() or CallSite(None, None, None)
--> 110         SparkContext._ensure_initialized(self, gateway=gateway)
    111         try:
    112             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

/Users/victor/Downloads/spark-1.4.1/python/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway)
    227         with SparkContext._lock:
    228             if not SparkContext._gateway:
--> 229                 SparkContext._gateway = gateway or launch_gateway()
    230                 SparkContext._jvm = SparkContext._gateway.jvm
    231 

/Users/victor/Downloads/spark-1.4.1/python/pyspark/java_gateway.pyc in launch_gateway()
     87                 callback_socket.close()
     88         if gateway_port is None:
---> 89             raise Exception("Java gateway process exited before sending the driver its port number")
     90 
     91         # In Windows, ensure the Java child processes do not linger after Python has exited.

Exception: Java gateway process exited before sending the driver its port number

[Carreau]

You should get this error if you get the wrong java (the 60M download instead of the 200+M download)

[vherasme]

I actually got jdk-8u60-macosx-x64.dmg which is 238.1 MB. Maybe I should restart the machine

[Carreau]

hum, I did not had to restart IIRC.