The ready condition of MachineDeployment doesn't change when a Machine CR is deleted

[jessehu]

What steps did you take and what happened:

1. Create a workload cluster (e.g. 3 CP + 3 Workers)
2. Delete a Machine CR related to MachineDeployment
3. MachineDeployment status doesn't change until the old Machine is deleted successfully and the new Machine CR is created.

What did you expect to happen:
The ready condition of MachineDeployment CR should change to False right after Machine CR is deleted. The Replicas or ReadyReplicas in MD.Status might be udpated as well as.

Anything else you would like to add:
When Delete a Machine CR related to KCP, the ready condition of KCP CR changes to False immediately (as expected). The Replicas, ReadyReplicas, Unavailable in MD.Status doesn't change until the new Machine CR is created.

Environment:

• Cluster-api version: v1.2.3
• minikube/kind version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[jessehu]

Related to PR #4625

[fabriziopandini]

/triage accepted

I have to look into this and come back with my consideration about the proposed changes, but I agree that the system should reflect changes as soon as possible and there should be consistency between KCP and MD

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[fabriziopandini]

/lifecycle frozen
/help
it will be great if someone can take over the research work behind this issue

[k8s-ci-robot]

@fabriziopandini:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/lifecycle frozen
/help
it will be great if someone can take over the research work behind this issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[haijianyang]

@fabriziopandini

When Machine being deleted, the associated Node is still ready(node.status.conditions is ready) although it is withNoSchedule taint. So the readyReplicas of MachineDeployment and MachineSet are not changed.
Only after the Machine is deleted, MachineSet and MachineDeployment will update readyReplicas.

machineset_controller knows whether the node is ready by IsNodeReady and set readyReplicas.

//  /cluster-api/controllers/noderefutil/util.go
// IsNodeReady returns true if a node is ready; false otherwise.
func IsNodeReady(node *corev1.Node) bool {
	if node == nil {
		return false
	}
	for _, c := range node.Status.Conditions {
		if c.Type == corev1.NodeReady {
			return c.Status == corev1.ConditionTrue
		}
	}
	return false
}

# mycluster-workergroup1-pnh2w is being deleted
kubectl get nodes
NAME                           STATUS                     ROLES                  AGE   VERSION
mycluster-controlplane-bcr8j   Ready                      control-plane,master   84m   v1.23.14
mycluster-controlplane-cdv2d   Ready                      control-plane,master   87m   v1.23.14
mycluster-controlplane-m5zjc   Ready                      control-plane,master   82m   v1.23.14
mycluster-workergroup1-pnh2w   Ready,SchedulingDisabled   <none>                 85m   v1.23.14
mycluster-workergroup1-pxm7m   Ready                      <none>                 84s   v1.23.14
mycluster-workergroup1-t5std   Ready                      <none>                 84m   v1.23.14

Currently, we cannot know whether any nodes in the cluster are being deleted(through MachineDeployment and MachineSet).

Can we add deletingReplicas to MachineDeployment and MachineSet to record the number of Machines being deleted. When Machine is deleted, MachineDeployment and MachineSet can update deletingReplicas in real time.

[haijianyang]

@fabriziopandini