Support spark.driver.extraJavaOptions

[kimoonkim]

Supports extra JVM options to pass to the driver command line. spark.driver.extraJavaOptions is an existing config key (documented here) and commonly used to pass additional JVM GC settings, etc.

[ash211]

This looks good @kimoonkim ! Out of curiosity, what flags are you needing to put on the driver JVM?

[mccheah]

Thanks for submitting this - also looks fine to me!

[ssuchter]

I’ve commonly seen people wanting to set various JVM memory options with -XX (yes, sometimes matters on the driver) and java.library.path things. Sean On January 25, 2017 at 3:07:10 PM, mccheah ([email protected]) wrote: Thanks for submitting this - also looks fine to me! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#48 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2E2zRrbYVSKrDRbiYo0Gw7gM-QyF-5ks5rV9WcgaJpZM4LuGhA> .

[mccheah]

I think spark.driver.memory is the canonical configuration for setting driver memory, and we do respect this in KubernetesSparkRestServer. But there's other more obscure things like GC settings that are worth overriding.

[ash211]

Let's merge when tests complete

[kimoonkim]

Thanks for the reviews!

@ash211 I was playing with -javaagent JVM option, to insert a JMX-based Spark metrics sync that can expose the driver metrics to prometheus. The idea is well described in this blog.

Note there is also a similar option for executors. I was hoping to explore that as well, but it seems the current code for launching executor pods does not manipulate the command line at runtime. So maybe next time :-)

[lins05]

+1 for this. I was thinking of how to attach a intellij debugger to the driver code, now i can make use of the extra driver java options.

[lins05]

Well done. I have successfully made the spark driver process attach to my intellij debug server, with something like this:

 --conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=n,suspend=y,address=myhost:5005