Skip to content
This repository was archived by the owner on Feb 4, 2021. It is now read-only.
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 69 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@

[![codecov](https://codecov.io/gh/mozilla/jupyter-spark/branch/master/graph/badge.svg)](https://codecov.io/gh/mozilla/jupyter-spark)

Jupyter Notebook extension for Apache Spark integration.
Jupyter Notebook extension and JupyterLab plugin for Apache Spark integration.

Includes a progress indicator for the current Notebook cell if it invokes a
Spark job. Queries the Spark UI service on the backend to get the required
Spark job information.
## Jupyter Notebook

In the classic Jupyter Notebook, it includes a progress indicator for the
current Notebook cell if it invokes a Spark job. It also provides a modal dialog
to show the progress of all running Spark jobs.

![Alt text](/screenshots/ProgressBar.png?raw=true "Spark progress bar")

Expand All @@ -19,12 +21,27 @@ button, or press ```Alt+S```.

![Alt text](/screenshots/Dialog.png?raw=true "Spark dialog")

## JupyterLab

In JupyterLab, it provides a left-side pane showing the currently running Spark
jobs.

![Alt text](/screenshots/JupyterLab.png?raw=true "Spark side pane")

## Server

The server that communicates between the Jupyter server and Spark is the same
regardless of the frontend used. It queries the Spark UI service on the backend
to get the required Spark job information.

A proxied version of the Spark UI can be accessed at
http://localhost:8888/spark.

## Installation

To install, simply run:
### Jupyter Notebook

To install for Jupyter Notebook, simply run:

```
pip install jupyter-spark
Expand All @@ -48,7 +65,7 @@ jupyter nbextension list
jupyter serverextension list
```

Pleaes feel free to install [lxml](http://lxml.de/) as well to improve
Please feel free to install [lxml](http://lxml.de/) as well to improve
performance of the server side communication to Spark using your favorite
package manager, e.g.:

Expand All @@ -74,6 +91,48 @@ jupyter nbextension disable --py jupyter_spark
jupyter nbextension uninstall --py jupyter_spark
pip uninstall jupyter-spark
```
### JupyterLab

To install for JupyterLab, simply run:

```
pip install jupyter-spark
jupyter serverextension enable --py jupyter_spark
jupyter labextension install jupyter_spark
```

To double-check if the extension was correctly installed run:

```
jupyter nbextension list
jupyter labextension list
```

Please feel free to install [lxml](http://lxml.de/) as well to improve
performance of the server side communication to Spark using your favorite
package manager, e.g.:

```
pip install lxml
```

For development and testing, clone the project and run from a shell in the
project's root directory:

```
pip install -e .
jupyter serverextension enable --py jupyter_spark
npm install
jupyter labextension install .
```

To uninstall the extension run:

```
jupyter serverextension disable --py jupyter_spark
jupyter labextension disable --py jupyter_spark
pip uninstall jupyter-spark
```

## Configuration

Expand All @@ -91,6 +150,10 @@ installation is working.

## Changelog

### 0.5.0

- Added support for JupyterLab

### 0.3.0 (2016-07-04)

- Rewrote proxy to use an async Tornado handler and HTTP client to fetch
Expand Down
28 changes: 28 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,32 @@
{
"private": true,
"name": "jupyter-spark",
"version": "0.4.0",
"description": "A Jupyter Notebook extension for Apache Spark integration",
"author": "Mozilla Firefox Data Platform",
"main": "src/jupyter_spark/static/jupyterlab-plugin.js",
"files": [
"src/jupyter_spark/static/extension.js",
"src/jupyter_spark/static/spark.css"
],
"keywords": [
"jupyter",
"jupyterlab",
"jupyterlab-extension"
],
"jupyterlab": {
"extension": true
},
"scripts": {},
"dependencies": {
"@jupyterlab/application": "^0.16.3",
"@jupyterlab/apputils": "^0.16.4",
"@jupyterlab/docregistry": "^0.16.3",
"@jupyterlab/notebook": "^0.16.3",
"@phosphor/disposable": "^1.1.2",
"jquery": "^3.3.1",
"bootstrap": "^4.1.1"
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

"devDependencies": {
"eslint": "^4.17.0",
"eslint-plugin-amd-imports": "^4.0.0"
Expand Down
Binary file added screenshots/JupyterLab.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading