java.net.UnknownHostException: kubernetes in KubernetesClusterSchedulerBackend for Minikube

[mccheah]

While working on getting Ingresses up in #70, I had to upgrade Minikube. I upgraded it to 0.16.0, and started seeing this error in KubernetesClusterSchedulerBackend:

java.net.UnknownHostException: kubernetes: Try again

Looks like setting the API server url to https://kubernetes doesn't work in this environment. We do this here: https://github.com/apache-spark-on-k8s/spark/blob/k8s-support-alternate-incremental/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/kubernetes/KubernetesClusterSchedulerBackend.scala#L47

[mccheah]

@foxish I think this was part of the Kubernetes client upgrade change. Can we use the master as defined by spark.master instead?

[mccheah]

Or we can look into why the DNS won't resolve properly.

[foxish]

I'll look into it shortly. I think we should figure out the DNS issue because it's definitely the correct way to contact the master from within the cluster.

…

On Feb 13, 2017 9:07 PM, "mccheah" ***@***.***> wrote: Or we can look into why the DNS won't resolve properly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#112 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3U5zDjeeeh2VlLW5Fwlkiw6NznM2gDks5rcTacgaJpZM4MAEIC> .

[foxish]

Also, using kubernetes.master fails when someone uses the local proxy to submit their driver, because the master URL localhost:port on the user's machine will be inaccessible from within the cluster. On Feb 13, 2017 9:10 PM, "Anirudh Ramanathan" <ramanathana@google.com> wrote:

…

I'll look into it shortly. I think we should figure out the DNS issue because it's definitely the correct way to contact the master from within the cluster. On Feb 13, 2017 9:07 PM, "mccheah" ***@***.***> wrote: > Or we can look into why the DNS won't resolve properly. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#112 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AA3U5zDjeeeh2VlLW5Fwlkiw6NznM2gDks5rcTacgaJpZM4MAEIC> > . >

[foxish]

@mccheah I just tried this and faced no issues with DNS resolution. However, it turns out minikube doesn't publish an ExternalIP for its node, but instead publishes a LegacyHostIP. I'm sending a PR to add that to the addresses we look at. More about at that kubernetes/kubernetes#9267

[mccheah]

Strange - what version of Minikube did you use? I upgraded to 0.16.0 on my machine.

[foxish]

I am using 0.16.0 as well. It's the one on Linux. Can you do 'kubectl get svc' on yours? Do you see a service named kubernetes?

[mccheah]

I do see a service named "kubernetes". Using minikube for mac here, and it runs on Virtualbox.

[mccheah]

I tried reverting my changes that I had locally to work on #70, and it still fails on the main k8s-support-alternate-incremental branch.

[foxish]

That is indeed very strange. Makes me think there is something wrong with DNS in your local cluster?

~./minikube-linux-amd64 version
minikube version: v0.16.0

~ kubectl get svc
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes   10.0.0.1     <none>        443/TCP    19d
svc          10.0.0.246   <none>        1000/TCP   18d

~ kubectl run -i -t busybox --image=busybox --restart=Never
Waiting for pod default/busybox to be running, status is Pending, pod ready: false
If you don't see a command prompt, try pressing enter.
/ # ping kubernetes
PING kubernetes (10.0.0.1): 56 data bytes

Mine resolves the master successfully from within the cluster. Can you try the above step with trying to resolve kubernetes from within a pod? It seems like it could be a minikube issue as well. I tried the same after removing my ~/.minikube directory. Same results.

[mccheah]

Taking out the ~/.minikube directory did not help. I deleted said directory, deleted the Minikube VM, and removed the Virtualbox network - to no avail. Looks like DNS is warped on my side somehow. Can someone else try running the tests on Mac with Minikube 0.16.0?

[foxish]

Can you resolve the kubernetes service from within a busybox pod?

[mccheah]

Looks like it can resolve, I get a similar result:

/ # ping kubernetes
PING kubernetes (10.0.0.1): 56 data bytes