Skip to content

[SPARK-22131][MESOS] Mesos driver secrets #19437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

[SPARK-22131][MESOS] Mesos driver secrets #19437

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b289bcc
[SPARK-22131] Mesos driver secrets. The driver launches executors tha…
susanxhuynh Oct 4, 2017
6f062c0
[SPARK-22131] Updated docs
susanxhuynh Oct 5, 2017
73b2cbf
Addressed review comments: (1) pluralized SECRET_xyz config entries, …
susanxhuynh Oct 9, 2017
770d307
Addressed review comments: added examples to docs, reverted the chang…
susanxhuynh Oct 18, 2017
b2a3675
Fixed formatting
susanxhuynh Oct 18, 2017
a801799
Mentioned need for custom module in file-based secrets docs; formatting.
susanxhuynh Oct 18, 2017
3cd1ae8
Addressed review comments: grouped properties together in the documen…
susanxhuynh Oct 25, 2017
88dfb42
Fixed formatting in docs table cell, style.
susanxhuynh Oct 26, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 89 additions & 22 deletions docs/running-on-mesos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -485,39 +485,106 @@ See the [configuration page](configuration.html) for information on Spark config
</tr>

<tr>
<td><code>spark.mesos.driver.secret.envkeys</code></td>
<td><code>(none)</code></td>
<td>
A comma-separated list that, if set, the contents of the secret referenced
by spark.mesos.driver.secret.names or spark.mesos.driver.secret.values will be
set to the provided environment variable in the driver's process.
<code>spark.mesos.driver.secret.values</code>,
Copy link
Contributor

@vanzin vanzin Oct 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you generate the docs to see how this looks? I wonder how browsers will try to render this extra-wide column.

Copy link
Contributor Author

@susanxhuynh susanxhuynh Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks okay in a browser. This column takes up about 25% of the page width. Here's a screenshot:
documentation screenshot

<code>spark.mesos.driver.secret.names</code>,
<code>spark.mesos.executor.secret.values</code>,
<code>spark.mesos.executor.secret.names</code>,
</td>
</tr>
<tr>
<td><code>spark.mesos.driver.secret.filenames</code></td>
<td><code>(none)</code></td>
<td>
A comma-separated list that, if set, the contents of the secret referenced by
spark.mesos.driver.secret.names or spark.mesos.driver.secret.values will be
written to the provided file. Paths are relative to the container's work
directory. Absolute paths must already exist. Consult the Mesos Secret
protobuf for more information.
<p>
A secret is specified by its contents and destination. These properties
specify a secret's contents. To specify a secret's destination, see the cell below.
</p>
<p>
You can specify a secret's contents either (1) by value or (2) by reference.
</p>
<p>
(1) To specify a secret by value, set the
<code>spark.mesos.[driver|executor].secret.values</code>
property, to make the secret available in the driver or executors.
For example, to make a secret password "guessme" available to the driver process, set:

<pre>spark.mesos.driver.secret.values=guessme</pre>
</p>
<p>
(2) To specify a secret that has been placed in a secret store
by reference, specify its name within the secret store
by setting the <code>spark.mesos.[driver|executor].secret.names</code>
property. For example, to make a secret password named "password" in a secret store
available to the driver process, set:

<pre>spark.mesos.driver.secret.names=password</pre>
</p>
<p>
Note: To use a secret store, make sure one has been integrated with Mesos via a custom
<a href="http://mesos.apache.org/documentation/latest/secrets/">SecretResolver
module</a>.
</p>
<p>
To specify multiple secrets, provide a comma-separated list:

<pre>spark.mesos.driver.secret.values=guessme,passwd123</pre>

or

<pre>spark.mesos.driver.secret.names=password1,password2</pre>
</p>
</td>
</tr>

<tr>
<td><code>spark.mesos.driver.secret.names</code></td>
<td><code>(none)</code></td>
<td>
A comma-separated list of secret references. Consult the Mesos Secret
protobuf for more information.
<code>spark.mesos.driver.secret.envkeys</code>,
<code>spark.mesos.driver.secret.filenames</code>,
<code>spark.mesos.executor.secret.envkeys</code>,
<code>spark.mesos.executor.secret.filenames</code>,
</td>
</tr>
<tr>
<td><code>spark.mesos.driver.secret.values</code></td>
<td><code>(none)</code></td>
<td>
A comma-separated list of secret values. Consult the Mesos Secret
protobuf for more information.
<p>
A secret is specified by its contents and destination. These properties
specify a secret's destination. To specify a secret's contents, see the cell above.
</p>
<p>
You can specify a secret's destination in the driver or
executors as either (1) an environment variable or (2) as a file.
</p>
<p>
(1) To make an environment-based secret, set the
<code>spark.mesos.[driver|executor].secret.envkeys</code> property.
The secret will appear as an environment variable with the
given name in the driver or executors. For example, to make a secret password available
to the driver process as $PASSWORD, set:

<pre>spark.mesos.driver.secret.envkeys=PASSWORD</pre>
</p>
<p>
(2) To make a file-based secret, set the
<code>spark.mesos.[driver|executor].secret.filenames</code> property.
The secret will appear in the contents of a file with the given file name in
the driver or executors. For example, to make a secret password available in a
file named "pwdfile" in the driver process, set:

<pre>spark.mesos.driver.secret.filenames=pwdfile</pre>
</p>
<p>
Paths are relative to the container's work directory. Absolute paths must
already exist. Note: File-based secrets require a custom
<a href="http://mesos.apache.org/documentation/latest/secrets/">SecretResolver
module</a>.
</p>
<p>
To specify env vars or file names corresponding to multiple secrets,
provide a comma-separated list:

<pre>spark.mesos.driver.secret.envkeys=PASSWORD1,PASSWORD2</pre>

or

<pre>spark.mesos.driver.secret.filenames=pwdfile1,pwdfile2</pre>
</p>
</td>
</tr>

Expand Down
64 changes: 35 additions & 29 deletions resource-managers/mesos/src/main/scala/org/apache/spark/deploy/mesos/config.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,39 @@ import org.apache.spark.internal.config.ConfigBuilder

package object config {

private[spark] class MesosSecretConfig private[config](taskType: String) {
private[spark] val SECRET_NAMES =
ConfigBuilder(s"spark.mesos.$taskType.secret.names")
.doc("A comma-separated list of secret reference names. Consult the Mesos Secret " +
"protobuf for more information.")
.stringConf
.toSequence
.createOptional

private[spark] val SECRET_VALUES =
ConfigBuilder(s"spark.mesos.$taskType.secret.values")
.doc("A comma-separated list of secret values.")
.stringConf
.toSequence
.createOptional

private[spark] val SECRET_ENVKEYS =
ConfigBuilder(s"spark.mesos.$taskType.secret.envkeys")
.doc("A comma-separated list of the environment variables to contain the secrets." +
"The environment variable will be set on the driver.")
.stringConf
.toSequence
.createOptional

private[spark] val SECRET_FILENAMES =
ConfigBuilder(s"spark.mesos.$taskType.secret.filenames")
.doc("A comma-separated list of file paths secret will be written to. Consult the Mesos " +
"Secret protobuf for more information.")
.stringConf
.toSequence
.createOptional
}

/* Common app configuration. */

private[spark] val SHUFFLE_CLEANER_INTERVAL_S =
Expand Down Expand Up @@ -64,36 +97,9 @@ package object config {
.stringConf
.createOptional

private[spark] val SECRET_NAME =
ConfigBuilder("spark.mesos.driver.secret.names")
.doc("A comma-separated list of secret reference names. Consult the Mesos Secret protobuf " +
"for more information.")
.stringConf
.toSequence
.createOptional

private[spark] val SECRET_VALUE =
ConfigBuilder("spark.mesos.driver.secret.values")
.doc("A comma-separated list of secret values.")
.stringConf
.toSequence
.createOptional
private[spark] val driverSecretConfig = new MesosSecretConfig("driver")

private[spark] val SECRET_ENVKEY =
ConfigBuilder("spark.mesos.driver.secret.envkeys")
.doc("A comma-separated list of the environment variables to contain the secrets." +
"The environment variable will be set on the driver.")
.stringConf
.toSequence
.createOptional

private[spark] val SECRET_FILENAME =
ConfigBuilder("spark.mesos.driver.secret.filenames")
.doc("A comma-seperated list of file paths secret will be written to. Consult the Mesos " +
"Secret protobuf for more information.")
.stringConf
.toSequence
.createOptional
private[spark] val executorSecretConfig = new MesosSecretConfig("executor")

private[spark] val DRIVER_FAILOVER_TIMEOUT =
ConfigBuilder("spark.mesos.driver.failoverTimeout")
Expand Down
138 changes: 26 additions & 112 deletions ...mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ import org.apache.mesos.{Scheduler, SchedulerDriver}
import org.apache.mesos.Protos.{TaskState => MesosTaskState, _}
import org.apache.mesos.Protos.Environment.Variable
import org.apache.mesos.Protos.TaskStatus.Reason
import org.apache.mesos.protobuf.ByteString

import org.apache.spark.{SecurityManager, SparkConf, SparkException, TaskState}
import org.apache.spark.deploy.mesos.MesosDriverDescription
Expand Down Expand Up @@ -394,39 +393,20 @@ private[spark] class MesosClusterScheduler(
}

// add secret environment variables
getSecretEnvVar(desc).foreach { variable =>
if (variable.getSecret.getReference.isInitialized) {
logInfo(s"Setting reference secret ${variable.getSecret.getReference.getName}" +
s"on file ${variable.getName}")
} else {
logInfo(s"Setting secret on environment variable name=${variable.getName}")
}
envBuilder.addVariables(variable)
MesosSchedulerBackendUtil.getSecretEnvVar(desc.conf, config.driverSecretConfig)
.foreach { variable =>
if (variable.getSecret.getReference.isInitialized) {
logInfo(s"Setting reference secret ${variable.getSecret.getReference.getName} " +
s"on file ${variable.getName}")
} else {
logInfo(s"Setting secret on environment variable name=${variable.getName}")
}
envBuilder.addVariables(variable)
}

envBuilder.build()
}

private def getSecretEnvVar(desc: MesosDriverDescription): List[Variable] = {
val secrets = getSecrets(desc)
val secretEnvKeys = desc.conf.get(config.SECRET_ENVKEY).getOrElse(Nil)
if (illegalSecretInput(secretEnvKeys, secrets)) {
throw new SparkException(
s"Need to give equal numbers of secrets and environment keys " +
s"for environment-based reference secrets got secrets $secrets, " +
s"and keys $secretEnvKeys")
}

secrets.zip(secretEnvKeys).map {
case (s, k) =>
Variable.newBuilder()
.setName(k)
.setType(Variable.Type.SECRET)
.setSecret(s)
.build
}.toList
}

private def getDriverUris(desc: MesosDriverDescription): List[CommandInfo.URI] = {
val confUris = List(conf.getOption("spark.mesos.uris"),
desc.conf.getOption("spark.mesos.uris"),
Expand All @@ -440,6 +420,23 @@ private[spark] class MesosClusterScheduler(
CommandInfo.URI.newBuilder().setValue(uri.trim()).setCache(useFetchCache).build())
}

private def getContainerInfo(desc: MesosDriverDescription): ContainerInfo.Builder = {
val containerInfo = MesosSchedulerBackendUtil.buildContainerInfo(desc.conf)

MesosSchedulerBackendUtil.getSecretVolume(desc.conf, config.driverSecretConfig)
.foreach { volume =>
if (volume.getSource.getSecret.getReference.isInitialized) {
logInfo(s"Setting reference secret ${volume.getSource.getSecret.getReference.getName} " +
s"on file ${volume.getContainerPath}")
} else {
logInfo(s"Setting secret on file name=${volume.getContainerPath}")
}
containerInfo.addVolumes(volume)
}

containerInfo
}

private def getDriverCommandValue(desc: MesosDriverDescription): String = {
val dockerDefined = desc.conf.contains("spark.mesos.executor.docker.image")
val executorUri = getDriverExecutorURI(desc)
Expand Down Expand Up @@ -579,89 +576,6 @@ private[spark] class MesosClusterScheduler(
.build
}

private def getContainerInfo(desc: MesosDriverDescription): ContainerInfo.Builder = {
val containerInfo = MesosSchedulerBackendUtil.containerInfo(desc.conf)

getSecretVolume(desc).foreach { volume =>
if (volume.getSource.getSecret.getReference.isInitialized) {
logInfo(s"Setting reference secret ${volume.getSource.getSecret.getReference.getName}" +
s"on file ${volume.getContainerPath}")
} else {
logInfo(s"Setting secret on file name=${volume.getContainerPath}")
}
containerInfo.addVolumes(volume)
}

containerInfo
}


private def getSecrets(desc: MesosDriverDescription): Seq[Secret] = {
def createValueSecret(data: String): Secret = {
Secret.newBuilder()
.setType(Secret.Type.VALUE)
.setValue(Secret.Value.newBuilder().setData(ByteString.copyFrom(data.getBytes)))
.build()
}

def createReferenceSecret(name: String): Secret = {
Secret.newBuilder()
.setReference(Secret.Reference.newBuilder().setName(name))
.setType(Secret.Type.REFERENCE)
.build()
}

val referenceSecrets: Seq[Secret] =
desc.conf.get(config.SECRET_NAME).getOrElse(Nil).map(s => createReferenceSecret(s))

val valueSecrets: Seq[Secret] = {
desc.conf.get(config.SECRET_VALUE).getOrElse(Nil).map(s => createValueSecret(s))
}

if (valueSecrets.nonEmpty && referenceSecrets.nonEmpty) {
throw new SparkException("Cannot specify VALUE type secrets and REFERENCE types ones")
}

if (referenceSecrets.nonEmpty) referenceSecrets else valueSecrets
}

private def illegalSecretInput(dest: Seq[String], s: Seq[Secret]): Boolean = {
if (dest.isEmpty) { // no destination set (ie not using secrets of this type
return false
}
if (dest.nonEmpty && s.nonEmpty) {
// make sure there is a destination for each secret of this type
if (dest.length != s.length) {
return true
}
}
false
}

private def getSecretVolume(desc: MesosDriverDescription): List[Volume] = {
val secrets = getSecrets(desc)
val secretPaths: Seq[String] =
desc.conf.get(config.SECRET_FILENAME).getOrElse(Nil)

if (illegalSecretInput(secretPaths, secrets)) {
throw new SparkException(
s"Need to give equal numbers of secrets and file paths for file-based " +
s"reference secrets got secrets $secrets, and paths $secretPaths")
}

secrets.zip(secretPaths).map {
case (s, p) =>
val source = Volume.Source.newBuilder()
.setType(Volume.Source.Type.SECRET)
.setSecret(s)
Volume.newBuilder()
.setContainerPath(p)
.setSource(source)
.setMode(Volume.Mode.RO)
.build
}.toList
}

/**
* This method takes all the possible candidates and attempt to schedule them with Mesos offers.
* Every time a new task is scheduled, the afterLaunchCallback is called to perform post scheduled
Expand Down
Loading