Regression introduced by NodePort PR

[ash211]

Just debugged with @mccheah , and we found an issue with the NodePort implementation of the jar upload process (#30)

The code assumes that the port on all returned k8s nodes is accessible, but we found in testing with my kubeadm-created cluster that actually the k8s master does NOT forward the traffic to the pod behind the service. Only the non-master k8s nodes do.

Setup:

myhost-16 is an apiserver, and myhost-17/18/19 are non-masters but part of the k8s cluster. Cluster assembled using https://kubernetes.io/docs/getting-started-guides/kubeadm/

response returned from jetty (the KubernetesSparkRestApi):
curl -XGET http://myhost-17.mydomain:30827/v1/submissions/ping
curl -XGET http://myhost-18.mydomain:30827/v1/submissions/ping
curl -XGET http://myhost-19.mydomain:30827/v1/submissions/ping

request times out:
curl -XGET http://myhost-16.mydomain:30827/v1/submissions/ping

Proposed solution: fail over in the liveness pings from non-responsive nodes to other nodes. This fixes the problem with the master, and more generally also nodes failing out of the cluster.

[foxish]

Only the pods running kube-proxy will expose the NodePort. It is possible for the master to run it but not common IIRC.

[ash211]

Introduced failover logic in #69 that makes this work (though failover is a little slow for my taste). Additional work to make this better happening in the two linked issues from issue 69.