showcase how to test operator

[scrocquesel]

Actually, tests are E2E integration tests and require external dependencies (database instance, k8s cluster). In some CI environment, it may not be possible to do so.

We should be able to unit test with mock for those dependencies. This would allow to add more tests as we have more control on the state of those dependencies.

See related works on quarkus extension to unit test mysql schema operator : https://github.com/quarkiverse/quarkus-operator-sdk/pull/201/files#diff-43e4c3f21bc716a010119d458691c0a57aeafe813c6606e805c7d63a62da3e17

[andreaTP]

I was looking at the provided E2E tests and I found a good foundation for what I was trying to build.
Specifically I'm completely happy to have "real" resources in the cluster (as opposed to mocks) as long as the lifecycle of such resources is managed by the testing infrastructure.

Those are the "missing features" I found:

• decide if the resources are created on each test or on each testing class (e.g. beforeEach/beforeAll)
• spin additional infrastructure and manage the lifecycle of it
• using a simple flag being able to run the operator locally (in-process) or remotely (in-kubernetes)

I just opened: #891 to show-case those.

[scrocquesel]

I was looking at the provided E2E tests and I found a good foundation for what I was trying to build. Specifically I'm completely happy to have "real" resources in the cluster (as opposed to mocks) as long as the lifecycle of such resources is managed by the testing infrastructure.

I agree even if I think that it works well for small E2E test campaign. When the number of tests augment, it become longer and longer to run them all. And usually, there are not so independent as unit tests and you can't just run one test method. Also, the arrange part can be tricky, especially for very edge cases.

What I'm not sure is what should we test the framework doesn't already guarantee us. I mean, all the logic of calling reconcile is done by the framework and should be well tested. Does unit testing reconcile method quicker and easier ?

In my case, the operator is actually operating the deployment of an application that need a lot of other external components and operators to be able to observe READY state. Not only, arranging the infrastructure is difficult but waiting to observe the result can take quite a long time. And all of those dependents component are already tested on their own. What should be pretty isolated starts to be a project on its own.

If the operator is just creating a deployment that start a pod in a millisecond, obviously it is pretty easy to test it will eventually succeed. But how do you test the case when the deployment failed due to some infrastructure errors (k8s quota reached, k8s api down, mysql down, etc...). Introducing chaos could be very challenging.

I'm not saying E2E are useless, they definitively need to exists. But what can be offered to a team developping a more complex operator to be able to test it without installing all the components of the real platform.

Those are the "missing features" I found:

• decide if the resources are created on each test or on each testing class (e.g. beforeEach/beforeAll)
• spin additional infrastructure and manage the lifecycle of it
• using a simple flag being able to run the operator locally (in-process) or remotely (in-kubernetes)

I just opened: #891 to show-case those.

I think a way to start/stop the operator between test to be able to arrange the infrastructure while it is not running would be needed as well.

[scrocquesel]

For an exemple of complexity, you can look at https://github.com/nais/naiserator/blob/master/pkg/synchronizer/synchronizer_test.go
It uses the go EnvTest which basically is just k8s api server and etcd, pretty much what fabricio mock server is offering.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[scrocquesel]

I think it can be closed, there is enough samples and the JUnit extension fo integration tests. And dependent resources allow for better unit testing.