Mismatch between ControlPlaneReady and Conditions.ControlPlaneReady

[knabben]

What steps did you take and what happened:
Running CAPD with tilt from HEAD, created a new runtime extension webhoook server and added the function hook for ControlPlaneInitialized, I wanted to execute a few steps AFTER the Cluster being ready, first try was to read from request.Cluster.Status.ControlPlaneReady boolean to check if I can create my pod in the WL cluster. This operations never happened even if the condition of Type ControlPlaneReady is True eventually.

E0820 17:11:02.074289      18 handlers.go:75] "Control plane not ready retrying."
phase Provisioned
controlplaneready false
infraready true
Ready False

// conditions
ControlPlaneInitialized True
ControlPlaneReady False
InfrastructureReady True

... repeats 8x

I0820 17:11:06.312747      18 handlers.go:55] "AfterControlPlaneInitialized is called."
E0820 17:11:06.312803      18 handlers.go:75] "Control plane not ready retrying."
phase Provisioned
controlplaneready false
infraready true

// conditions
Ready True
ControlPlaneInitialized True
ControlPlaneReady True
InfrastructureReady True

I0820 17:11:06.344126      18 handlers.go:55] "AfterControlPlaneInitialized is called."

In the last line request.Cluster.ControlPlaneReady == false, and all Conditions are true, as noted this does not happen to the InfrastructureReady.

Can this be for the fact I'm returning an ReponseStatusFailure while waiting for the cp be ready?

What did you expect to happen:
To both fields be in sync when the cp is ready.

Environment:

• Cluster-api version: HEAD
• minikube/kind version: 0.14.0
• Kubernetes version: (use kubectl version): 1.24
• OS (e.g. from /etc/os-release): Debian (WSL)

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[killianmuldoon]

These two fields are set slightly differently, with ControlPlaneReady being based on the status.ready field in the Control Plane object and the condition being based on the conditions in the Control Plane object.

I think, though, there's a couple of questions I have with the approach here, so it would be good to understand the use case.

1. Are you returning a failure from the AfterControlPlaneInitialized hook to prevent reconciliation from continuing?
2. Why are you waiting for ControlPlaneReady? What doesn't work at AfterControlPlaneInitialized that works with your addtional condition?

[knabben]

These two fields are set slightly differently, with ControlPlaneReady being based on the status.ready field in the Control Plane object and the condition being based on the conditions in the Control Plane object.

I think, though, there's a couple of questions I have with the approach here, so it would be good to understand the use case.

1. Are you returning a failure from the AfterControlPlaneInitialized hook to prevent reconciliation from continuing?
2. Why are you waiting for ControlPlaneReady? What doesn't work at AfterControlPlaneInitialized that works with your addtional condition?

1. that's correct. I should be retrying here, not sure how to do that with hookd response object
2. I'm using the cache tracker when using aftercpinitialized directly the connection returns but it's not usable, when using the config on a clientset it returns an empty list of pods for example (maybe I was missing an err somewhere), the fact is it works normally after waiting for the ready condition.

[sbueringer]

cc @ykakarap

[sbueringer]

If I see correctly AfterControlPlaneInitialized is called once ControlPlaneInitializedCondition is true.

cluster-api/internal/controllers/topology/cluster/reconcile_state.go

Line 209 in 0b48593

if isControlPlaneInitialized(s.Current.Cluster) {

ControlPlaneInitializedCondition is set to true once we have a control plane machine with a nodeRef:

cluster-api/internal/controllers/cluster/cluster_controller.go

Lines 497 to 500 in bbe7c2f

	if util.IsControlPlaneMachine(m) && m.Status.NodeRef != nil {
	conditions.MarkTrue(cluster, clusterv1.ControlPlaneInitializedCondition)
	return ctrl.Result{}, nil
	}

That nodeRef is only set if we were able to get a Node object from the workload cluster (via a client form ClusterCacheTracker)

cluster-api/internal/controllers/machine/machine_controller_noderef.go

Lines 59 to 93 in bbe7c2f

	remoteClient, err := r.Tracker.GetClient(ctx, util.ObjectKey(cluster))
	if err != nil {
	return ctrl.Result{}, err
	}
	// Even if Status.NodeRef exists, continue to do the following checks to make sure Node is healthy
	node, err := r.getNode(ctx, remoteClient, providerID)
	if err != nil {
	if err == ErrNodeNotFound {
	// While a NodeRef is set in the status, failing to get that node means the node is deleted.
	// If Status.NodeRef is not set before, node still can be in the provisioning state.
	if machine.Status.NodeRef != nil {
	conditions.MarkFalse(machine, clusterv1.MachineNodeHealthyCondition, clusterv1.NodeNotFoundReason, clusterv1.ConditionSeverityError, "")
	return ctrl.Result{}, errors.Wrapf(err, "no matching Node for Machine %q in namespace %q", machine.Name, machine.Namespace)
	}
	conditions.MarkFalse(machine, clusterv1.MachineNodeHealthyCondition, clusterv1.NodeProvisioningReason, clusterv1.ConditionSeverityWarning, "")
	// No need to requeue here. Nodes emit an event that triggers reconciliation.
	return ctrl.Result{}, nil
	}
	log.Error(err, "Failed to retrieve Node by ProviderID")
	r.recorder.Event(machine, corev1.EventTypeWarning, "Failed to retrieve Node by ProviderID", err.Error())
	return ctrl.Result{}, err
	}
	// Set the Machine NodeRef.
	if machine.Status.NodeRef == nil {
	machine.Status.NodeRef = &corev1.ObjectReference{
	Kind: node.Kind,
	APIVersion: node.APIVersion,
	Name: node.Name,
	UID: node.UID,
	}
	log.Info("Infrastructure provider reporting spec.providerID, Kubernetes node is now available", machine.Spec.InfrastructureRef.Kind, klog.KRef(machine.Spec.InfrastructureRef.Namespace, machine.Spec.InfrastructureRef.Name), "providerID", providerID, "node", klog.KRef("", machine.Status.NodeRef.Name))
	r.recorder.Event(machine, corev1.EventTypeNormal, "SuccessfulSetNodeRef", machine.Status.NodeRef.Name)
	}

I could be overlooking something, but looks to me like the workload cluster apiserver has to be reachable before the hook is called.

[killianmuldoon]

After doing a little bit of testing here there is a significant drift between the two markers depending on network configuration:

The condition ControlPlaneReady is based on whether or not the ControlPlane machines and components are created and ready.

Cluster .status.controlPlaneReady, is a summary of whether or not the Nodes are in a ready state. It will never become true until the CNI is in place in practice. i.e. NetworkReady for the node object must be true.

I'm not sure of the different networking constraints for different providers, but I can imagine a situation where no CNI means limited API server access.

We might want to align these values (or possibly deprecate the old status fields). I'm not read into the full context here, but is there any reason to keep the status 'Ready' fields around (vs conditions) in the long term? i.e. apart from backward compatibility etc.

We could deprecate without an intention to remove until the next API revision to signal that the condition is the better value to rely on.

[sbueringer]

I'm not sure of the different networking constraints for different providers, but I can imagine a situation where no CNI means limited API server access.

I'm not sure if that is possible. apiserver should not depend on CNI, otherwise it would be a deadlock (at least for "external access" like from the mgmt cluster).

One other data point. Our quickstart depends on the APIserver being reachable so that you can deploy CNI at the end of the provisioning. A similar workflow is the default for kubeadm afaik.

[killianmuldoon]

I'm not sure if that is possible. apiserver should not depend on CNI, otherwise it would be a deadlock (at least for "external access" like from the mgmt cluster).

Agreed for most situations (there's enough edge cases here that I'm sure there's some way this could work out.

If this is the case should our API SERVER AVAILABLE column in kubectl for KCP reflect the condition rather than status.ready as it does today?

Right now it's telling us if the nodes are ready and is different from ControlPlaneInitialized and ControlPlaneReady.

I think the drift between these two is a definite weakness in the API

[sbueringer]

If this is the case should our API SERVER AVAILABLE column in kubectl for KCP reflect the condition rather than status.ready as it does today?

Is this possible with CRD columns? Their feature set is rather limited.

[killianmuldoon]

Seems like it should be possible with jsonPath - kubectl get cluster -o custom-columns='READY:status.conditions[0].status' works from kubectl and I guess it should work for additionalPrinterColumns too?

I think the important question though is whether to try to close the gap between these two values. Right now the API has a couple of values with near-identical names but different semantics, and in some of the cases they are seemingly wrong based on their names and lead to bad assumptions by consumers.

[sbueringer]

Seems like it should be possible with jsonPath - kubectl get cluster -o custom-columns='READY:status.conditions[0].status' works from kubectl and I guess it should work for additionalPrinterColumns too?

I'm not sure if CRD columns support the full jsonPath. It's probably also not a good idea to depend on a certain condition being the first one in the array (not sure if more dynamic array element matching is supported by CRD columns). I had some problems with CRD columns in the past.

We might want to align these values (or possibly deprecate the old status fields). I'm not read into the full context here, but is there any reason to keep the status 'Ready' fields around (vs conditions) in the long term? i.e. apart from backward compatibility etc.

IIRC the plan was to migrate eventually to conditions. I'm not sure if we wanted to drop the bools accordingly.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten