[SPARK-29070][CORE] Make SparkLauncher log full spark-submit command line #25777
Conversation
There was a problem hiding this comment.
I think you can just reflect this method and call sparkLauncher.createBuilder().command() as described in the mailing list. Not sure how much useful this API is in general yet.
There was a problem hiding this comment.
How about simply logging the command from SparkLauncher instead of adding a new public method, then?
There was a problem hiding this comment.
How is it differernt from setting SPARK_PRINT_LAUNCH_COMMAND to 1?
There was a problem hiding this comment.
That prints the command to stderr, which is not always easy to access (depending on the environment, etc.). Using the logging API gives clients more flexibility in controlling where the information appears (they can use the full power of log4j, for example).
jeff303
force-pushed
the
SPARK-29070
branch
2 times, most recently
from
73121ef to
047d74d
Compare
jeff303
changed the title
| */ | ||
| public class SparkLauncher extends AbstractLauncher<SparkLauncher> { | ||
|
|
||
| // use JUL logger, relying on jul-to-slf4j bridge |
dev/.rat-excludes
Outdated
| vote.tmpl | ||
| SessionManager.java | ||
| SessionHandler.java | ||
| .python-version |
There was a problem hiding this comment.
It seems odd to have this here. What is generating this file? It's definitely not being added by this PR. And other builds are not failing. So I think this deserves an explanation...
There was a problem hiding this comment.
It's a pyenv metadata file. I was using pyenv to manage Python versions locally for purposes of running the PySpark tests. It doesn't really need a license file since it's just a single line indicating the Python version. Nonetheless, it indeed has nothing to do with this change, so I'll remove it.
|
ok to test |
…line Log the full spark-submit command in SparkSubmit#launchApplication, using a JUL logger
jeff303
force-pushed
the
SPARK-29070
branch
from
047d74d to
0af4f88
Compare
|
Test build #111444 has finished for PR 25777 at commit
|
|
Test build #111442 has finished for PR 25777 at commit
|
|
retest this please |
|
Test build #111453 has finished for PR 25777 at commit
|
|
Merging to master. |
5 participants
Log the full spark-submit command in SparkSubmit#launchApplication
Adding .python-version (pyenv file) to RAT exclusion list
What changes were proposed in this pull request?
Original motivation here, expanded in the Jira.. In essence, we want to be able to log the full
spark-submitcommand being constructed bySparkLauncherWhy are the changes needed?
Currently, it is not possible to directly obtain this information from the
SparkLauncherinstance, which makes debugging and customer support more difficult.Does this PR introduce any user-facing change?
No
How was this patch tested?
coresbttests were executed. TheSparkLauncherSuite(where I added assertions to an existing test) was also checked. Within that,testSparkLauncherGetErroris failing, but that appears not to have been caused by this change (failing for me even on the parent commit of c18f849).