Skip to content

[SPARK-25887][K8S] Configurable K8S context support #22904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-25887][K8S] Configurable K8S context support #22904

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7fa5097
[SPARK-25887][K8S] Configurable K8S context support
rvesse Oct 31, 2018
8fc744d
[SPARK-25887][K8S] Clean up issues
rvesse Dec 3, 2018
f8b0632
[SPARK-25887][K8S] Avoid possible NPE
rvesse Dec 4, 2018
c76f752
[SPARK-25887][K8S] Address PR comments
rvesse Dec 4, 2018
5e4c340
Fix scalastyle errors
rvesse Dec 19, 2018
7832480
[SPARK-25887][K8S] Simplify logging statement
rvesse Jan 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions docs/running-on-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,16 @@ the Spark application.

## Kubernetes Features

### Configuration File

Your Kubernetes config file typically lives under `.kube/config` in your home directory or in a location specified by the `KUBECONFIG` environment variable. Spark on Kubernetes will attempt to use this file to do an initial auto-configuration of the Kubernetes client used to interact with the Kubernetes cluster. A variety of Spark configuration properties are provided that allow further customising the client configuration e.g. using an alternative authentication method.
Copy link
Contributor

@aditanase aditanase Nov 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add more details about what configuration we can reuse from the context. For example, would spark.master be taken from kubernetes master? I'm trying to understand what's the net new benefit for somebody using this in their day-to-day work.

Also, now that we have client mode, I anticipate many users switching to that to emulate a typical Mesos deployment (e.g. driver in client mode, kept alive by Marathon). I am assuming this does not apply in client mode, where all the options need to be explicitly specified.

Copy link
Member Author

@rvesse rvesse Nov 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be frank I'm not really sure why there are different config options for client vs cluster mode and this may have changed with some of the cleanup that @vanzin has been doing lately to simplify the configuration code.

Personally I have never needed to use any of the additional configuration properties in either client/cluster mode as the auto-configuration from my K8S config file has always been sufficient. At worst I've needed to set KUBECONFIG to select the correct config file for the cluster I want to submit to.

Note that the core behaviour (the auto-configuration) has always existed implicitly in the K8S backend but was just not called out explicitly previously in the docs. This PR primarily just makes it more explicit and flexible for users who have multiple contexts in their config files.

WRT spark.master Spark in general requires that to always be set and will use that to override whatever is present in the K8S config file regardless.

Copy link
Contributor

@aditanase aditanase Dec 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I just opened https://issues.apache.org/jira/browse/SPARK-26295 and @vanzin redirected me to this thread. Would love your eyes on that issue, see if we can use your work here to close that too.

In short, If there is code that propagates the kube context along this path, I'm not aware of it, would love to see some documentation:

laptop with kubectl and context > k apply -f spark-driver-client-mode.yaml -> deployment starts 1 instance of driver pod in arbitrary namespace -> spark submit from start.sh inside the docker container -> ...

There is no kubectl or "kube context" in the docker container, it's just the spark distro and my jars. So where would the driver pod get the account from?

PS: agreed that there are too many config options on the auth side, maybe we could consolidate them more.


### Contexts

Kubernetes configuration files can contain multiple contexts that allow for switching between different clusters and/or user identities. By default Spark on Kubernetes will use your current context (which can be checked by running `kubectl config current-context`) when doing the initial auto-configuration of the Kubernetes client.

In order to use an alternative context users can specify the desired context via the Spark configuration property `spark.kubernetes.context` e.g. `spark.kubernetes.context=minikube`.

### Namespaces

Kubernetes has the concept of [namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).
Expand Down Expand Up @@ -406,13 +416,23 @@ Some of these include:

# Configuration

See the [configuration page](configuration.html) for information on Spark configurations. The following configurations are
specific to Spark on Kubernetes.
See the [configuration page](configuration.html) for information on Spark configurations. The following configurations are specific to Spark on Kubernetes.

#### Spark Properties

<table class="table">
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr>
<td><code>spark.kubernetes.context</code></td>
<td><code>(none)</code></td>
<td>
The context from the user Kubernetes configuration file used for the initial
auto-configuration of the Kubernetes client library. When not specified then
the users current context is used. <strong>NB:</strong> Many of the
auto-configured settings can be overridden by the use of other Spark
configuration properties e.g. <code>spark.kubernetes.namespace</code>.
</td>
</tr>
<tr>
<td><code>spark.kubernetes.namespace</code></td>
<td><code>default</code></td>
Expand Down
12 changes: 12 additions & 0 deletions resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,18 @@ import org.apache.spark.internal.config.ConfigBuilder

private[spark] object Config extends Logging {

val KUBERNETES_CONTEXT =
ConfigBuilder("spark.kubernetes.context")
.doc("The desired context from your K8S config file used to configure the K8S " +
"client for interacting with the cluster. Useful if your config file has " +
"multiple clusters or user identities defined. The client library used " +
"locates the config file via the KUBECONFIG environment variable or by defaulting " +
"to .kube/config under your home directory. If not specified then your current " +
"context is used. You can always override specific aspects of the config file " +
"provided configuration using other Spark on K8S configuration options.")
.stringConf
.createOptional

val KUBERNETES_NAMESPACE =
ConfigBuilder("spark.kubernetes.namespace")
.doc("The namespace that will be used for running the driver and executor pods.")
Expand Down
19 changes: 13 additions & 6 deletions ...rnetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,21 @@ import java.io.File
import com.google.common.base.Charsets
import com.google.common.io.Files
import io.fabric8.kubernetes.client.{ConfigBuilder, DefaultKubernetesClient, KubernetesClient}
import io.fabric8.kubernetes.client.Config.autoConfigure
import io.fabric8.kubernetes.client.utils.HttpClientUtils
import okhttp3.Dispatcher

import org.apache.spark.SparkConf
import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.internal.Logging
import org.apache.spark.util.ThreadUtils

/**
* Spark-opinionated builder for Kubernetes clients. It uses a prefix plus common suffixes to
* parse configuration keys, similar to the manner in which Spark's SecurityManager parses SSL
* options for different components.
*/
private[spark] object SparkKubernetesClientFactory {
private[spark] object SparkKubernetesClientFactory extends Logging {

def createKubernetesClient(
master: String,
Expand All @@ -42,9 +44,6 @@ private[spark] object SparkKubernetesClientFactory {
sparkConf: SparkConf,
defaultServiceAccountToken: Option[File],
defaultServiceAccountCaCert: Option[File]): KubernetesClient = {

// TODO [SPARK-25887] Support configurable context

val oauthTokenFileConf = s"$kubernetesAuthConfPrefix.$OAUTH_TOKEN_FILE_CONF_SUFFIX"
val oauthTokenConf = s"$kubernetesAuthConfPrefix.$OAUTH_TOKEN_CONF_SUFFIX"
val oauthTokenFile = sparkConf.getOption(oauthTokenFileConf)
Expand All @@ -67,8 +66,16 @@ private[spark] object SparkKubernetesClientFactory {
val dispatcher = new Dispatcher(
ThreadUtils.newDaemonCachedThreadPool("kubernetes-dispatcher"))

// TODO [SPARK-25887] Create builder in a way that respects configurable context
val config = new ConfigBuilder()
// Allow for specifying a context used to auto-configure from the users K8S config file
val kubeContext = sparkConf.get(KUBERNETES_CONTEXT).filter(_.nonEmpty)
logInfo(s"Auto-configuring K8S client using " +
kubeContext.map("context " + _).getOrElse("current context") +
s" from users K8S config file")

// Start from an auto-configured config with the desired context
// Fabric 8 uses null to indicate that the users current context should be used so if no
// explicit setting pass null
val config = new ConfigBuilder(autoConfigure(kubeContext.getOrElse(null)))
Copy link
Contributor

@vanzin vanzin Dec 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens here when the context does not exist? Does it fall back to the default?

e.g. in cluster mode, the config you're adding will be propagated to the driver, and then this code will be called with the same context as the submission node. What if that context does not exist inside the driver container?

Copy link
Member Author

@rvesse rvesse Dec 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the context does not exist then Fabric 8 falls back to other ways of auto-configuring itself (e.g. service account)

Fabric 8 skips any file based auto-configuration if there is no K8S config file present (https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L436-L459).

Since we don't propagate the submission clients config file into the driver pods no auto-configuration from config file will be attempted in the driver because there won't be a config file present.

Copy link
Contributor

@vanzin vanzin Dec 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't propagate the submission clients config file

We don't propagate the config file itself, but we do propagate all its contents, as far as I remember. But given your explanation it should work fine, unless there's a different config context with the same name inside the container...

Copy link
Contributor

@vanzin vanzin Dec 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Just to clarify I'm referring to the Spark config. Too many config files involved here...)

Copy link
Member Author

@rvesse rvesse Dec 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes and I was referring to the K8S config file :) And yes the fact that we would propagate spark.kubernetes.context into the pod shouldn't be an issue because there won't be any K8S config file for it to interact with inside the pod as in-pod K8S config should be from the service account token that gets injected into the pod

Copy link
Contributor

@vanzin vanzin Dec 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this enhancement does not apply to client mode

If you mean "client mode inside a k8s-managed docker container", then yes, you may need to do extra stuff, like mount the appropriate credentials. But in the "client mode with driver inside k8s pod" case, Spark does not create that pod for you. So I'm not sure how Spark can help with anything there; the serviceName configuration seems targeted at propagating the credentials of the submitter to the driver pod, and in that case Spark is not creating the driver pod at all.

Copy link
Contributor

@aditanase aditanase Dec 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this is starting to sink in. I just realized that in my deployment scenario, all I need to do is to specify the serviceAccountName on my pod template that contains the driver.

That leaves me confused though - what other "client mode deployments" do you see happening on K8S? What does client mode mean to you?
Also - how should one interpret this paragraph in the docs?
In client mode, use spark.kubernetes.authenticate.serviceAccountName instead.

Copy link
Contributor

@vanzin vanzin Dec 10, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does client mode mean to you?

Client mode means that the driver process / container is not started by Spark. It's started directly by the user.

Also - how should one interpret this paragraph in the docs?

I have no idea.

Copy link
Contributor

@aditanase aditanase Dec 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course :) I think what I'm asking is what real-world options do we forsee for client mode in a k8s context. I don't see many beyond a pod that the user creates. Whether straight spark or something like jupyter/zeppelin notebook, at the end of the day it's probably a docker container living in a pod.

Thanks for taking the time, I won't spam this thread anymore. I'll continue the discussion on the issue I created, which I'll repurpose for improving the docs in this area.

Copy link
Member Author

@rvesse rvesse Dec 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aditanase One practical use case is interactive Spark code where the Spark REPL (and driver) is running at some login shell to which the user has access to e.g. on a physical edge node of the cluster and the actual Spark executors are running as K8S pods. This is something we have customers using today.

.withApiVersion("v1")
.withMasterUrl(master)
.withWebsocketPingInterval(0)
Expand Down