[SPARK-26600] Update spark-submit usage message #23518
Conversation
| | Spark on YARN and K8S only: | ||
| | --principal PRINCIPAL Principal to be used to login to KDC. | ||
| | --keytab KEYTAB The full path to the file that contains the keytab for the | ||
| | principal specified above. |
There was a problem hiding this comment.
what happens to the rest of the original text?
There was a problem hiding this comment.
Thanks for reviewing. The text that I cut is mostly about the implementation of --keytab (i.e. spark.kerberos.keytab) and refers to the YARN implementation. In K8S the implementation is different. Also on YARN this has evolved recently BTW. I think it is safe and shorter just to omit the implementation details in this scope.
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
Outdated
Show resolved
Hide resolved
|
Test build #4511 has started for PR 23518 at commit |
|
Test build #4515 has finished for PR 23518 at commit
|
|
Test build #4516 has started for PR 23518 at commit |
|
Test build #4518 has started for PR 23518 at commit |
jerryshao
left a comment
jerryshao
left a comment
There was a problem hiding this comment.
LGTM. Is this the only place needs to be changed?
|
I would say so, as I am now aware of other places in the code or doc listing the options for spark-submit with the same details provided in spark-submit --help message. As for the point of clarifying the use of "--keytab" option for Kubernetes, I see that SPARK-26595 takes care of it with an update in docs/security.md |
|
Test build #4519 has finished for PR 23518 at commit
|
|
Merged to master |
## What changes were proposed in this pull request? Spark-submit usage message should be put in sync with recent changes in particular regarding K8S support. These are the proposed changes to the usage message: --executor-cores NUM -> can be useed for Spark on YARN and K8S --principal PRINCIPAL and --keytab KEYTAB -> can be used for Spark on YARN and K8S --total-executor-cores NUM> can be used for Spark standalone, YARN and K8S In addition this PR proposes to remove certain implementation details from the --keytab argument description as the implementation details vary between YARN and K8S, for example. ## How was this patch tested? Manually tested Closes apache#23518 from LucaCanali/updateSparkSubmitArguments. Authored-by: Luca Canali <[email protected]> Signed-off-by: Sean Owen <[email protected]>
6 participants
What changes were proposed in this pull request?
Spark-submit usage message should be put in sync with recent changes in particular regarding K8S support. These are the proposed changes to the usage message:
--executor-cores NUM -> can be useed for Spark on YARN and K8S
--principal PRINCIPAL and --keytab KEYTAB -> can be used for Spark on YARN and K8S
--total-executor-cores NUM> can be used for Spark standalone, YARN and K8S
In addition this PR proposes to remove certain implementation details from the --keytab argument description as the implementation details vary between YARN and K8S, for example.
How was this patch tested?
Manually tested