Skip to content

[SPARK-26600] Update spark-submit usage message #23518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-26600] Update spark-submit usage message #23518

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b5d03ee
update spark-submit usage message
LucaCanali Jan 11, 2019
930f242
Changed K8S with Kubernetes.
LucaCanali Jan 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 11 additions & 12 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -576,27 +576,26 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
| --kill SUBMISSION_ID If given, kills the driver specified.
| --status SUBMISSION_ID If given, requests the status of the driver specified.
|
| Spark standalone and Mesos only:
| Spark standalone, Mesos and Kubernetes only:
| --total-executor-cores NUM Total cores for all executors.
|
| Spark standalone and YARN only:
| --executor-cores NUM Number of cores per executor. (Default: 1 in YARN mode,
| or all available cores on the worker in standalone mode)
| Spark standalone, YARN and Kubernetes only:
| --executor-cores NUM Number of cores used by each executor. (Default: 1 in
| YARN and K8S modes, or all available cores on the worker
| in standalone mode).
|
| YARN-only:
| Spark on YARN and Kubernetes only:
| --principal PRINCIPAL Principal to be used to login to KDC.
| --keytab KEYTAB The full path to the file that contains the keytab for the
| principal specified above.
Copy link
Member

@felixcheung felixcheung Jan 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens to the rest of the original text?

Copy link
Contributor Author

@LucaCanali LucaCanali Jan 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing. The text that I cut is mostly about the implementation of --keytab (i.e. spark.kerberos.keytab) and refers to the YARN implementation. In K8S the implementation is different. Also on YARN this has evolved recently BTW. I think it is safe and shorter just to omit the implementation details in this scope.

|
| Spark on YARN only:
| --queue QUEUE_NAME The YARN queue to submit to (Default: "default").
| --num-executors NUM Number of executors to launch (Default: 2).
| If dynamic allocation is enabled, the initial number of
| executors will be at least NUM.
| --archives ARCHIVES Comma separated list of archives to be extracted into the
| working directory of each executor.
| --principal PRINCIPAL Principal to be used to login to KDC, while running on
| secure HDFS.
| --keytab KEYTAB The full path to the file that contains the keytab for the
| principal specified above. This keytab will be copied to
| the node running the Application Master via the Secure
| Distributed Cache, for renewing the login tickets and the
| delegation tokens periodically.
""".stripMargin
)

Expand Down