Spark configuration options with spaces aren't parsed properly in default images

[mccheah]

This is most notorious with spark.driver.extraJavaOptions and spark.executor.extraJavaOptions. The following configuration won't work properly:

--conf spark.driver.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails

Because we pass the driver's JVM options in a single environment variable, we don't handle the spaces in a single configuration value properly.

[ash211]

ok I think we have a way to do this:

• in the launcher, split the spark.*.extraJavaOptions variables using Utils.splitCommandString
• add a series of environment variables to each pod, named EXTRA_JAVA_OPT_0, EXTRA_JAVA_OPT_1, EXTRA_JAVA_OPT_2, etc.
• in the Dockerfile, read these variables out of the environment and apply to the Java command, possibly using bash arrays (requires bash 4+) to avoid an unbounded listing of variables

[erikerlandson]

Is there a reason these sequences don't work when encased in quotes?

--conf spark.driver.extraJavaOptions="-XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails"

(provided the quotes are re-propagated as needed)

[mccheah]

I don't know why that isn't working, though this format is something I tried. I think it has something to do with the nuances of bash environment variable substitution.

[erikerlandson]

It bugs me that this doesn't work, but OTOH I can't definitively suggest a fix.

You're saying that putting quotes around it like:
${JAVA_HOME}/bin/java "$SPARK_DRIVER_JAVA_OPTS" ...
wasn't working?

[ash211]

correct, just adding those quotes to the Dockerfile was insufficient to get the tests to pass (we spent most of Thursday wrestling with this)

[mccheah]

If quotes are placed around the entire string then the entire string is passed as a single argument to the JVM. That doesn't parse out and delimit the options properly.

[erikerlandson]

Maybe something like:

echo $SPARK_DRIVER_JAVA_OPTS | readarray -d ' ' -t TEMP_OPTS
${JAVA_HOME}/bin/java "${TEMP_OPTS[@]}" ...

That might at least remove the need for the separate env vars and the internal processing?

[mccheah]

the -d option isn't supported by Bash as installed by our Docker images in alpina linux. I also think that this would fail because we want -Dspark.executor.extraJavaOptions=-Dopt1=opt1Value -Dopt2=opt2Value to be treated as a single JVM option.

[erikerlandson]

I suppose we might work around the -d by doing something like

echo $SPARK_DRIVER_JAVA_OPTS | tr ' ' '\n' | readarray -t TEMP_OPTS

I'm a bit confused by the single-jvm-option consideration. I thought the problem with "$SPARK_DRIVER_JAVA_OPS" was that it presented multiple options as a single option when it shouldn't.

[mccheah]

This command: echo $SPARK_DRIVER_JAVA_OPTS | tr ' ' '\n' | readarray -t TEMP_OPTS - this suffers from the same issue for piping to readarray, in that the resulting array is always empty:

bash-4.3# export SPARK_DRIVER_JAVA_OPTS="-Dspark.app.name=spark -Dspark.executor.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError -XX:PrintGCDetails"
bash-4.3# echo $SPARK_DRIVER_JAVA_OPTS | tr ' ' '\n' | readarray -t TEMP_OPTS
bash-4.3# echo ${TEMP_OPTS[@]}

[mccheah]

(the blank line at the end is the output of echo which shows that the array is empty)

[tangzhankun]

I've also encountered issues related to space characters in conf options when I set custom environments. So this issue should be solved thoughtfully.
During my test,

1. If one set value with space in spark.app.name, the error happens before launching any container. I havn't time to debug it.
2. If one set value with space("my value1") in spark.executorEnv.key1, the error happens with "Could not find or load main class value1".
   The root cause is the wrong final java command line was constructed like below:

java ... -Dspark.executorEnv.key1=my value1 -D...

value1 was parsed by java as a main class name so failed to run.