Allow providing an OAuth token for authenticating against k8s #180
Conversation
|
@foxish @robert3005 for review. I don't quite know how to test this. Does Minikube support authentication with OAuth tokens instead of certificate and key files? I need this for an internal application but it seems like a good idea to have this option in general. The other question is whether or not the token contents should be provided directly in the Spark configuration or if the token configurations should be passed through a file. I chose the latter because it seems more secure. The |
|
Hmm, setting this on spark conf sounds like asking for trouble to me. I am not certain if there's a better place though. There's some sanitisation available in current release but the users need to enable it. |
|
Well in this change we're not passing the token itself, but only the location on the submitter's disk. That way the token itself isn't ever logged anywhere. |
robert3005
left a comment
robert3005
left a comment
There was a problem hiding this comment.
Can we maybe send the file across to the pods maybe instead of sparkconf?
| """.stripMargin) | ||
| .stringConf | ||
| .createOptional | ||
|
|
| } | ||
| sparkConf.get(KUBERNETES_OAUTH_TOKEN_FILE).foreach { f => | ||
| val oauthTokenFile = new File(f) | ||
| require(oauthTokenFile.isFile, s"OAuth token file provided at $f does not exist or is" + |
There was a problem hiding this comment.
Can we move the whole message to the new line and not break it?
|
Right, sorry I misread some other comment. Looks good now |
|
The more I think about it the more I think it makes sense to be allowed to pass the token value via Spark configuration instead of through a file. The reason is that it allows users of the We can erase the token value from the Spark configuration after we use it. |
|
@foxish is this reasonable? |
|
@mccheah, the fabric8 kubernetes-client has support for using OAuth tokens instead of client-certs which is what we use against GKE, if the .kube/config file is setup appropriately. Do we need this explicitly as an option to Spark submit if fabric8 handles it for us? |
|
It is useful to be able to provide the token dynamically instead of relying on a static configuration file. Multiple applications being submitted from the same host but from different users is an example. |
|
My .kube/config contains many clusters and a mechanism by which I am authorized to talk to them. The way I switch which cluster to talk to is using https://kubernetes.io/docs/user-guide/kubectl/kubectl_config_use-context/. If we expect a user to get stdout/stderr using |
|
One example use case is that the application submitter does not itself have the credentials, but rather is provided the credentials by some other party and the submitter wishes to forward those credentials through to spark-submit. |
|
That sounds reasonable. LGTM. |
|
@robert3005 good to merge? |
|
Yup.
…On Mon, 13 Mar 2017, 16:02 mccheah, ***@***.***> wrote:
@robert3005 <https://github.com/robert3005> good to merge?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#180 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAfQVMmcZWxpfxvcMFhBFq30w1nUzBR2ks5rlaDGgaJpZM4MXS8k>
.
|
* Allow providing an OAuth token for authenticating against k8s * Organize imports * Fix style * Remove extra newline * Use OAuth token data instead of a file. (cherry picked from commit 1aba361)
* Allow providing an OAuth token for authenticating against k8s * Organize imports * Fix style * Remove extra newline * Use OAuth token data instead of a file. (cherry picked from commit 1aba361)
…-spark-on-k8s#180) * Allow providing an OAuth token for authenticating against k8s * Organize imports * Fix style * Remove extra newline * Use OAuth token data instead of a file. (cherry picked from commit 1aba361) (cherry picked from commit 35724a3)
…-spark-on-k8s#180) * Allow providing an OAuth token for authenticating against k8s * Organize imports * Fix style * Remove extra newline * Use OAuth token data instead of a file. (cherry picked from commit 1aba361)
Closes #179