Run tests against any specified k8s cluster

[ash211]

We currently run integration tests only against a minikube cluster that the integration tests specify.

I'd propose we split that out so integration tests can be run against any specified k8s cluster (likely via kubectl config) without assuming minikube. We can still keep the code that launches the minikube cluster though and provide docs for the two-step process: 1) start minikube 2) run tests against it.

This also allows us to more easily run against various cloud providers container environments. Possibly in the future we could even run integration tests against N cloud providers using donated compute time from those providers. I think Apache Spark has some tests that run in Amazon AWS on donated time (rumor).

[mccheah]

The Minikube startup was originally tied to the execution of the tests because it is not obvious how to get the Maven reactor build to trigger the starting and stopping of Minikube before and after running the tests. The problem with making it a two-step process is that it wouldn't be automated in Travis CI.

The ideal way I see going about doing this is to expose a system property for setting the Kubernetes master URL, set the system property to match Minikube's in the default Maven build, and have Maven start up the Minikube cluster as well as run the integration tests in the default execution of build/mvn integration-test. The developer can override the system property in their own testing using -Dmaster=value from the command line or equivalent settings in their IDE.

But I'm not familiar enough with Maven to know how to tie build steps to execution of commands, so the easiest way was just to integrate the cluster management into the test code itself.

[ssuchter]

When you say “tie build steps to execution of commands” - do you mean how to run specific external commands (e.g. do a fork/exec) at particular points in the maven execution? That’s generally done with the maven-exec-plugin. But is that what you were asking?

[mccheah]

Yep, precisely that. The maven exec plugin perhaps is the way to go here.

Another point is that Minikube's IP address isn't always fixed. I've seen cases where consecutive startups of the Minikube machine results in different IP addresses, generally an increment on the last digit (100, 101, 102, etc). Thus if we were to start Minikube from the Maven reactor we would need to propagate the IP it was assigned to the test runtime.

[lins05]

The problem with making it a two-step process is that it wouldn't be automated in Travis CI.

@mccheah I don't think we can run minikube on travis. IIUC minikube launches a virtual machine, but travis builds are running inside a vm.

A google search leads to me travis-ci/travis-ci#6060:

We do not support running VirtualBox inside any of our compute environments and currently have no plans to support it. 

[foxish]

This item becomes even more important if minikube is unsupported on travis. I think we may still be able to get a "local-cluster" working on travis. This means using the script https://github.com/kubernetes/kubernetes/blob/master/hack/local-up-cluster.sh on a checked out release-branch of k8s, which runs a dockerized cluster.

[foxish]

I think the "local" cluster will need some additional effort to make kube-dns work. It might not be worth the effort. We can discuss this tomorrow, about how we want to setup the e2es here.

[cvpatel]

@mccheah I don't think we can run minikube on travis. IIUC minikube launches a virtual machine, but travis builds are running inside a vm.

A google search leads to me travis-ci/travis-ci#6060:

We do not support running VirtualBox inside any of our compute environments and currently have no plans to support it.

Yes, VirtualBox is not supported in Travis; but I've been trying to see if KVM driver could be used instead, but it doesn't work simply out-of-the box. There are folks would have reported varying degrees of success based on the environment, but mostly with 32-bit version. IIUC minikube only-supports 64-bit and requires VT-x/AMD-v (vmx), which probably won't be happening for travis (travis-ci/travis-ci#1419).

This item becomes even more important if minikube is unsupported on travis. I think we may still be able to get a "local-cluster" working on travis. This means using the script https://github.com/kubernetes/kubernetes/blob/master/hack/local-up-cluster.sh on a checked out release-branch of k8s, which runs a dockerized cluster.

But, if this works then it would be better for Travis, as docker is a fully supported service. @ssuchter this maybe another alternative in addition to the others that were discussed earlier for integration tests.

[foxish]

I was giving this some more thought. @mccheah @ash211, what do you think about removing the minikube dependencies from our integration tests, and instead relying on a property that we pass to the integration tests that lets us specify a context & kubernetes-master to run the tests against?

The documentation can link to minikube instructions in case someone wants to run the integration tests using it, but I feel that we shouldn't rely on the fragile string parsing we currently use around minikube. Even though the pre-integration-test phase is convenient, it is a leaky abstraction.

[foxish]

One thing this may need is an official "latest" docker image pushed to some public registry. Minikube circumvents this by creating the image on the node's docker daemon, but we cannot rely on that mechanism if we use any other platform.

[foxish]

Closing this, because we have this capability now as of #243