[WIP] Use the dependency download init-containers on executors. #249
Conversation
| val resolvedDriverJavaOpts = resolvedSparkConf.getAll.map { case (confKey, confValue) => | ||
| s"-D$confKey=$confValue" | ||
| sparkConfExecutorsFetchRemoteDeps.set( | ||
| EXECUTOR_RESOLVED_MOUNTED_CLASSPATH, resolvedLocalClassPath) |
There was a problem hiding this comment.
I'm not entirely sure if this is the best way to pass this information along. The submission client here has already resolved what the local classpath should be and assumes that the same localized classpath would be valid on the executors. This is similar to what the YARN submission client does, but it would be nicer to have the driver itself be responsible for determining the classpath of its executors. However, perhaps having such logic on the driver is repetitive if the submission client had already determined what the resolved classpath would be.
There was a problem hiding this comment.
Or we can just add jarsDownloadPath/* to the classpath. Everywhere in Spark the exact jar files are enumerated, but since we're guaranteeing that the download path is an empty directory volume mount and is strictly being used for jars, it might make sense to just reference the whole directory in this context.
mccheah
changed the title
|
Still missing some unit tests but the idea is ready for review - cc @foxish @erikerlandson . Dependent on #246. |
|
Superceded by #251. |
This makes it such that submitted jars are ahead on the classpath when the executor starts up.