Skip to content

Commit 233c214

Browse files
jeff303Marcelo Vanzin
authored andcommitted
[SPARK-29070][CORE] Make SparkLauncher log full spark-submit command line
Log the full spark-submit command in SparkSubmit#launchApplication Adding .python-version (pyenv file) to RAT exclusion list ### What changes were proposed in this pull request? Original motivation [here](http://apache-spark-user-list.1001560.n3.nabble.com/Is-it-possible-to-obtain-the-full-command-to-be-invoked-by-SparkLauncher-td35144.html), expanded in the [Jira](https://issues.apache.org/jira/browse/SPARK-29070).. In essence, we want to be able to log the full `spark-submit` command being constructed by `SparkLauncher` ### Why are the changes needed? Currently, it is not possible to directly obtain this information from the `SparkLauncher` instance, which makes debugging and customer support more difficult. ### Does this PR introduce any user-facing change? No ### How was this patch tested? `core` `sbt` tests were executed. The `SparkLauncherSuite` (where I added assertions to an existing test) was also checked. Within that, `testSparkLauncherGetError` is failing, but that appears not to have been caused by this change (failing for me even on the parent commit of c18f849). Closes #25777 from jeff303/SPARK-29070. Authored-by: Jeff Evans <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>
1 parent 420abb4 commit 233c214

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,11 @@
2626
import java.util.Map;
2727
import java.util.concurrent.ThreadFactory;
2828
import java.util.concurrent.atomic.AtomicInteger;
29+
import java.util.logging.Level;
30+
import java.util.logging.Logger;
2931

3032
import static org.apache.spark.launcher.CommandBuilderUtils.*;
33+
import static org.apache.spark.launcher.CommandBuilderUtils.join;
3134

3235
/**
3336
* Launcher for Spark applications.
@@ -38,6 +41,8 @@
3841
*/
3942
public class SparkLauncher extends AbstractLauncher<SparkLauncher> {
4043

44+
private static final Logger LOG = Logger.getLogger(SparkLauncher.class.getName());
45+
4146
/** The Spark master. */
4247
public static final String SPARK_MASTER = "spark.master";
4348

@@ -363,6 +368,9 @@ public SparkAppHandle startApplication(SparkAppHandle.Listener... listeners) thr
363368

364369
String loggerName = getLoggerName();
365370
ProcessBuilder pb = createBuilder();
371+
if (LOG.isLoggable(Level.FINE)) {
372+
LOG.fine(String.format("Launching Spark application:%n%s", join(" ", pb.command())));
373+
}
366374

367375
boolean outputToLog = outputStream == null;
368376
boolean errorToLog = !redirectErrorStream && errorStream == null;

0 commit comments

Comments
 (0)