0.15.0 install hang

[kfox1111]

In a new minikube instance (1.10.1), I installed the crd's and that works. but applying the olm manifest is hanging:

[kfox@zathras ~]$ kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/$OLMVER/olm.yaml
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com configured

Tried to download it and apply it locally. same effect. Blew away minikube, started a fresh one, and same problem.

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

[kfox1111]

Not sure why, but i was thinking the crd's shouldn't be in that file either. there is a separate crd file usually.

[kfox1111]

if I cut out the crds for subscriptions and csv's out of the file, it loads ok.

[kfox1111]

If I cut everything but subscripts crd out, it fails to load.

[fabstu]

Same problem here, did not finish after running a whole night. It is my first time trying out olm and I didn't realize the release was only 1-2 hours old at the time^^ Gonna use 0.14.1 for the time being.

Platform: Digitalocean/Kubernetes

customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com configured

[kfox1111]

The manifest also refers to:

quay.io/operator-framework/olm@sha256:788915592f42882559a63c39e14ef211f64c93f31983ea67136533c7983f2896

which is inaccessible.

[kfox1111]

Still not sure what's going on with the CRDs. I think it has to do with there being the string '---' being used in the crd definitions. Stripping them works:

minikube start
kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.0/crds.yaml
curl https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.0/olm.yaml -L -s > /tmp/olm.yaml
sed -i 's/--- //g' /tmp/olm.yaml
kubectl apply -f /tmp/olm.yaml

Oddly, if you apply a second time, it hangs:

kubectl apply -f /tmp/olm.yaml

[kfox1111]

Ok. Narrowing it down I think...
The s/--- // thing is only semi-related.
if you simply do a:

kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.15.0/crds.yaml

Twice, it fails. Two of the crd's don't like being applied a second time. As its both in the crds.yaml and the olm.yaml (incorrect), it fails during olm.yaml unless there is some change.

I suspect some interaction with these crd's and the k8s serverside apply code in 1.18.

When loading the crd's into k8s, the apiserver throws this in the logs:

E0521 10:59:11.014017       1 customresource_handler.go:652] error building openapi models for subscriptions.operators.coreos.com: ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.volumes.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.volumes.items.<array>.properties.emptyDir.properties.sizeLimit has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.Subscription.properties.spec.properties.config.properties.volumes.items.<array>.properties.projected.properties.sources.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
I0521 10:59:11.015411       1 client.go:361] parsed scheme: "endpoint"
I0521 10:59:11.015443       1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379  <nil> 0 <nil>}]
E0521 10:59:11.115451       1 customresource_handler.go:652] error building openapi models for clusterserviceversions.operators.coreos.com: ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.strategy.properties.rollingUpdate.properties.maxSurge has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.strategy.properties.rollingUpdate.properties.maxUnavailable has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.postStart.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.postStart.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.preStop.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.lifecycle.properties.preStop.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.startupProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.containers.items.<array>.properties.startupProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.postStart.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.postStart.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.preStop.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.lifecycle.properties.preStop.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.startupProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.ephemeralContainers.items.<array>.properties.startupProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.env.items.<array>.properties.valueFrom.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.postStart.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.postStart.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.preStop.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.lifecycle.properties.preStop.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.resources.properties.limits.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.resources.properties.requests.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.startupProbe.properties.httpGet.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.initContainers.items.<array>.properties.startupProbe.properties.tcpSocket.properties.port has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.overhead.additionalProperties.schema has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.volumes.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.volumes.items.<array>.properties.emptyDir.properties.sizeLimit has invalid property: anyOf
ERROR $root.definitions.com.coreos.operators.v1alpha1.ClusterServiceVersion.properties.spec.properties.install.properties.spec.properties.deployments.items.<array>.properties.spec.properties.template.properties.spec.properties.volumes.items.<array>.properties.projected.properties.sources.items.<array>.properties.downwardAPI.properties.items.items.<array>.properties.resourceFieldRef.properties.divisor has invalid property: anyOf

[anik120]

We are aware that there are issues with the 0.15.0 release, and are working on recalling the 0.15.0 release and cutting a new release 0.15.1. Will update this issue when 0.15.1 is ready.

Apologies for the confusion.
/assign

[estroz]

@kfox1111 @anik120 0.15.0 seems to install fine for me now.

[kfox1111]

@anik120 +1. I'd be very curious what is going on with the CRD's. I've not seen a kubectl apply deadlock a client before.

@estroz With the workarounds above, I was able to start it now too.

[anik120]

@kfox1111 the issue was that the 0.15.0 tag in quay was deleted. It's been restored now, however the 0.15.0 assets contains a commit beyond the 0.15.0 tag in quay. We're publishing a patch before we recall 0.15.0 so that the assets and the quay tag are in sync.

[kfox1111]

That explains why it wouldn't start after the crds were removed from the olm.yaml file. Still unsure why the crd loading themselves ffail on reapply though. I'm wondering if 0.15.1 will have the same issue.

[anik120]

@kfox1111 the failure you're seeing on reapplying the CRDs might be due to a kubectl bug in this library: evanphx/json-patch@v4.4.0...v4.6.0

Applying quickstart/olm.yaml was failing because the CRDs were being included in that file too.
Thank you for bringing it to our notices. We're making sure we don't run into this for 0.15.1.