Skip to content

[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server #19047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server #19047

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
172fc20
[SPARK-21503]: Fixed the issue
Jul 21, 2017
81422e0
[SPARK-21503][UI]: Adding changes as per comments
Jul 25, 2017
55c6c37
Merge pull request #1 from apache/master
pgandhi999 Jul 26, 2017
f454c89
[SPARK-21541]: Spark Logs show incorrect job status for a job that do…
Jul 26, 2017
6b7d5c6
Revert "[SPARK-21541]: Spark Logs show incorrect job status for a job…
Jul 26, 2017
bc41664
Merge pull request #2 from apache/master
pgandhi999 Jul 28, 2017
e46126f
[SPARK-21503]- Making Changes as per comments
Jul 28, 2017
9b3cebc
Merge pull request #4 from apache/master
pgandhi999 Aug 1, 2017
7f03341
[SPARK-21503]: Reverting Unit Test Code
Aug 1, 2017
2d01cab
Merge pull request #5 from apache/master
pgandhi999 Aug 24, 2017
eaf63e6
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config…
Aug 24, 2017
e421a03
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config…
Aug 25, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions conf/spark-env.sh.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y")
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
# - SPARK_DAEMON_CLASSPATH, to set the classpath for all daemons
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers

# Generic options for the daemons used in the standalone deploy mode
Expand Down
4 changes: 4 additions & 0 deletions docs/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@ The history server can be configured as follows:
<td><code>SPARK_DAEMON_JAVA_OPTS</code></td>
<td>JVM options for the history server (default: none).</td>
</tr>
<tr>
<td><code>SPARK_DAEMON_CLASSPATH</code></td>
<td>Classpath for the history server (default: none).</td>
</tr>
<tr>
<td><code>SPARK_PUBLIC_DNS</code></td>
<td>
Expand Down
2 changes: 2 additions & 0 deletions docs/running-on-mesos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,8 @@ If you like to run the `MesosClusterDispatcher` with Marathon, you need to run t
The `MesosClusterDispatcher` also supports writing recovery state into Zookeeper. This will allow the `MesosClusterDispatcher` to be able to recover all submitted and running containers on relaunch. In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env by configuring `spark.deploy.recoveryMode` and related spark.deploy.zookeeper.* configurations.
For more information about these configurations please refer to the configurations [doc](configurations.html#deploy).

You can also specify any additional jars required by the `MesosClusterDispatcher` in the classpath by setting the environment variable SPARK_DAEMON_CLASSPATH in spark-env.

From the client, you can submit a job to Mesos cluster by running `spark-submit` and specifying the master URL
to the URL of the `MesosClusterDispatcher` (e.g: mesos://dispatcher:7077). You can view driver statuses on the
Spark cluster Web UI.
Expand Down
4 changes: 4 additions & 0 deletions docs/spark-standalone.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ You can optionally configure the cluster further by setting environment variable
<td><code>SPARK_DAEMON_JAVA_OPTS</code></td>
<td>JVM options for the Spark master and worker daemons themselves in the form "-Dx=y" (default: none).</td>
</tr>
<tr>
<td><code>SPARK_DAEMON_CLASSPATH</code></td>
<td>Classpath for the Spark master and worker daemons themselves (default: none).</td>
</tr>
<tr>
<td><code>SPARK_PUBLIC_DNS</code></td>
<td>The public DNS name of the Spark master and workers (default: none).</td>
Expand Down
5 changes: 5 additions & 0 deletions launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,16 +53,19 @@ public List<String> buildCommand(Map<String, String> env)
case "org.apache.spark.deploy.master.Master":
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
javaOptsKeys.add("SPARK_MASTER_OPTS");
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
memKey = "SPARK_DAEMON_MEMORY";
break;
case "org.apache.spark.deploy.worker.Worker":
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
javaOptsKeys.add("SPARK_WORKER_OPTS");
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
memKey = "SPARK_DAEMON_MEMORY";
break;
case "org.apache.spark.deploy.history.HistoryServer":
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
javaOptsKeys.add("SPARK_HISTORY_OPTS");
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
memKey = "SPARK_DAEMON_MEMORY";
break;
case "org.apache.spark.executor.CoarseGrainedExecutorBackend":
Expand All @@ -77,11 +80,13 @@ public List<String> buildCommand(Map<String, String> env)
break;
case "org.apache.spark.deploy.mesos.MesosClusterDispatcher":
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
break;
case "org.apache.spark.deploy.ExternalShuffleService":
case "org.apache.spark.deploy.mesos.MesosExternalShuffleService":
javaOptsKeys.add("SPARK_DAEMON_JAVA_OPTS");
javaOptsKeys.add("SPARK_SHUFFLE_OPTS");
extraClassPath = getenv("SPARK_DAEMON_CLASSPATH");
memKey = "SPARK_DAEMON_MEMORY";
break;
default:
Expand Down