Skip to content

🐛 Fix validation of worker topology names in Cluster resource #12069

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

🐛 Fix validation of worker topology names in Cluster resource #12069

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
242cebd
Add tests that fail when a worker topology name causes invalid Kubern…
dlipovetsky Apr 7, 2025
0079c98
Add validation that ensures worker topology names are valid Kubernete…
dlipovetsky Apr 7, 2025
efe9687
fixup! Add validation that ensures worker topology names are valid Ku…
dlipovetsky Apr 9, 2025
fbc09ec
fixup! Add validation that ensures worker topology names are valid Ku…
dlipovetsky Apr 9, 2025
2a930cc
fixup! Add tests that fail when a worker topology name causes invalid…
dlipovetsky Apr 9, 2025
70beeda
fixup! Add tests that fail when a worker topology name causes invalid…
dlipovetsky Apr 10, 2025
c206457
fixup! Add validation that ensures worker topology names are valid Ku…
dlipovetsky Apr 11, 2025
2ad7c26
fixup! Add tests that fail when a worker topology name causes invalid…
dlipovetsky Apr 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 48 additions & 6 deletions internal/topology/check/compatibility.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,8 +276,9 @@ func MachinePoolClassesAreUnique(clusterClass *clusterv1.ClusterClass) field.Err
return allErrs
}

// MachineDeploymentTopologiesAreValidAndDefinedInClusterClass checks that each MachineDeploymentTopology name is not empty
// and unique, and each class in use is defined in ClusterClass.spec.Workers.MachineDeployments.
// MachineDeploymentTopologiesAreValidAndDefinedInClusterClass checks that each MachineDeploymentTopology name is not empty,
// is a valid Kubernetes resource name, is a valid label value, and is unique. It also checks that each class in use is defined
// in ClusterClass.spec.Workers.MachineDeployments.
func MachineDeploymentTopologiesAreValidAndDefinedInClusterClass(desired *clusterv1.Cluster, clusterClass *clusterv1.ClusterClass) field.ErrorList {
var allErrs field.ErrorList
if desired.Spec.Topology.Workers == nil {
Expand All @@ -290,14 +291,34 @@ func MachineDeploymentTopologiesAreValidAndDefinedInClusterClass(desired *cluste
machineDeploymentClasses := mdClassNamesFromWorkerClass(clusterClass.Spec.Workers)
names := sets.Set[string]{}
for i, md := range desired.Spec.Topology.Workers.MachineDeployments {
// The Name must be a valid Kubernetes resource name, because it is used to generate the MachineDeployment name.
if errs := validation.IsDNS1123Subdomain(md.Name); len(errs) != 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JoelSpeed @sbueringer wondering if we should move (or duplicate) those validation on the API as a CRD markers

cluster-api/api/core/v1beta2/cluster_types.go

Line 694 in 7a22509

Name string `json:"name"`

cluster-api/api/core/v1beta2/cluster_types.go

Line 808 in 7a22509

Name string `json:"name"`

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to read through this thread first for context: #12069 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we were already had some validation for this field in code, and I could add a failing unit test to demonstrate the problem, I decided to modify the code.

However, I think we can express this validation in our API.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cluster-api/api/ipam/v1beta2/ipaddress_types.go

Lines 52 to 57 in d1ed8dc

// name of the IPAddressClaim.
// name must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character.
// +required
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=253
// +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`

Is an example of the same validation in openapi, personally I prefer CEL matching rules here as we can provide better validation feedback, but I believe we skipped on that for the moment because the errors returned were not including the invalid value

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we skipped on that for the moment because the errors returned were not including the invalid value

I'll wait for @sbueringer to add his thoghts.

I'm fine with CEL, code, or both for now, and removing code in the future. Just let me know what you'd like, so we can finish up here 😄

Copy link
Member

@sbueringer sbueringer Jul 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer CEL matching rules here as we can provide better validation feedback, but I believe we skipped on that for the moment because the errors returned were not including the invalid value

We skipped on CEL at the moment because based on Joel this would require a minimum v1.33 Kubernetes version on the mgmt cluster: #12235 (comment) :)

I'm tracking this here: #8190 (not sure if actually v1.33 or v1.34 because of the compatibility version thing in Kubernetes that ensures apiserver minor version rollbacks are possible).

We already have a Pattern on e.g. topology.classRef.name (on main: https://github.com/kubernetes-sigs/cluster-api/blob/main/api/core/v1beta2/cluster_types.go#L578). So it would be nice for consistency to do the same here.

The downsides of having this in OpenAPI are that we have to move the test coverage to envtest but maybe let's just do that. We have to do it sooner or later with CEL anyway. fyi, we already have tests like this in internal/webhooks/test, we can just add the new ones there as well.

If I understand correctly the combination of the IsDNS1123Subdomain pattern + MaxLength=63 makes the IsValidLabelValue validation redundant and we could just drop the code from the webhook? (and move test cases for that over to envtest as well)

P.S. Sorry for the delay
P.S.2 Let's please rebase onto main. Just easier to look at everything in its latest state

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly the combination of the IsDNS1123Subdomain pattern + MaxLength=63 makes the IsValidLabelValue validation redundant and we could just drop the code from the webhook? (and move test cases for that over to envtest as well)

I had also reached this conclusion

Copy link
Member

@sbueringer sbueringer Jul 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had also reached this conclusion

(Daniel also mentioned it explicitly in his godoc comment :))

for _, err := range errs {
allErrs = append(
allErrs,
field.Invalid(
field.NewPath("spec", "topology", "workers", "machineDeployments").Index(i).Child("name"),
md.Name,
fmt.Sprintf("must be a valid resource name: %s", err),
),
)
}
}

// The Name must also be a valid label value, because it is used in some label values.
//
// NOTE This check always returns true in practice, because OpenAPI validation in the
// Cluster CRD ensures that md.Name is <= 63 characters, and the IsDNS1123Subdomain check
// accepts a smaller set of characters than IsValidLabelValue. We keep this check to be able
// to unit test this function.
if errs := validation.IsValidLabelValue(md.Name); len(errs) != 0 {
for _, err := range errs {
allErrs = append(
allErrs,
field.Invalid(
field.NewPath("spec", "topology", "workers", "machineDeployments").Index(i).Child("name"),
md.Name,
fmt.Sprintf("must be a valid label value %s", err),
fmt.Sprintf("must be a valid label value: %s", err),
),
)
}
Expand Down Expand Up @@ -340,8 +361,9 @@ func MachineDeploymentTopologiesAreValidAndDefinedInClusterClass(desired *cluste
return allErrs
}

// MachinePoolTopologiesAreValidAndDefinedInClusterClass checks that each MachinePoolTopology name is not empty
// and unique, and each class in use is defined in ClusterClass.spec.Workers.MachinePools.
// MachinePoolTopologiesAreValidAndDefinedInClusterClass checks that each MachinePoolTopology name is not empty,
// is a valid Kubernetes resource name, is a valid label value, and is unique. It also checks that each class in use is defined
// in ClusterClass.spec.Workers.MachinePools.
func MachinePoolTopologiesAreValidAndDefinedInClusterClass(desired *clusterv1.Cluster, clusterClass *clusterv1.ClusterClass) field.ErrorList {
var allErrs field.ErrorList
if desired.Spec.Topology.Workers == nil {
Expand All @@ -354,14 +376,34 @@ func MachinePoolTopologiesAreValidAndDefinedInClusterClass(desired *clusterv1.Cl
machinePoolClasses := mpClassNamesFromWorkerClass(clusterClass.Spec.Workers)
names := sets.Set[string]{}
for i, mp := range desired.Spec.Topology.Workers.MachinePools {
// The Name must be a valid Kubernetes resource name, because it is used to generate the MachineDeployment name.
if errs := validation.IsDNS1123Subdomain(mp.Name); len(errs) != 0 {
for _, err := range errs {
allErrs = append(
allErrs,
field.Invalid(
field.NewPath("spec", "topology", "workers", "machinePools").Index(i).Child("name"),
mp.Name,
fmt.Sprintf("must be a valid resource name: %s", err),
),
)
}
}

// The Name must also be a valid label value, because it is used in some label values.
//
// NOTE This check always returns true in practice, because OpenAPI validation in the
// Cluster CRD ensures that md.Name is <= 63 characters, and the IsDNS1123Subdomain check
// accepts a smaller set of characters than IsValidLabelValue. We keep this check to be able
// to unit test this function.
if errs := validation.IsValidLabelValue(mp.Name); len(errs) != 0 {
for _, err := range errs {
allErrs = append(
allErrs,
field.Invalid(
field.NewPath("spec", "topology", "workers", "machinePools").Index(i).Child("name"),
mp.Name,
fmt.Sprintf("must be a valid label value %s", err),
fmt.Sprintf("must be a valid label value: %s", err),
),
)
}
Expand Down
120 changes: 120 additions & 0 deletions internal/topology/check/compatibility_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1333,6 +1333,66 @@ func TestMachineDeploymentTopologiesAreUniqueAndDefinedInClusterClass(t *testing
Build(),
wantErr: true,
},
{
name: "fail if MachineDeploymentTopology name is not a valid Kubernetes resource name",
clusterClass: builder.ClusterClass(metav1.NamespaceDefault, "class1").
WithInfrastructureClusterTemplate(
builder.InfrastructureClusterTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithControlPlaneTemplate(
builder.ControlPlane(metav1.NamespaceDefault, "cp1").Build()).
WithControlPlaneInfrastructureMachineTemplate(
builder.InfrastructureMachineTemplate(metav1.NamespaceDefault, "cpinfra1").Build()).
WithWorkerMachineDeploymentClasses(
*builder.MachineDeploymentClass("aa").
WithInfrastructureTemplate(
builder.InfrastructureMachineTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithBootstrapTemplate(
builder.BootstrapTemplate(metav1.NamespaceDefault, "bootstrap1").Build()).
Build()).
Build(),
cluster: builder.Cluster(metav1.NamespaceDefault, "cluster1").
WithTopology(
builder.ClusterTopology().
WithClass("class1").
WithVersion("v1.22.2").
WithMachineDeployment(
builder.MachineDeploymentTopology("under_score").
WithClass("aa").
Build()).
Build()).
Build(),
wantErr: true,
},
{
name: "fail if MachineDeploymentTopology name is not a valid Kubernetes label value",
clusterClass: builder.ClusterClass(metav1.NamespaceDefault, "class1").
WithInfrastructureClusterTemplate(
builder.InfrastructureClusterTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithControlPlaneTemplate(
builder.ControlPlane(metav1.NamespaceDefault, "cp1").Build()).
WithControlPlaneInfrastructureMachineTemplate(
builder.InfrastructureMachineTemplate(metav1.NamespaceDefault, "cpinfra1").Build()).
WithWorkerMachineDeploymentClasses(
*builder.MachineDeploymentClass("aa").
WithInfrastructureTemplate(
builder.InfrastructureMachineTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithBootstrapTemplate(
builder.BootstrapTemplate(metav1.NamespaceDefault, "bootstrap1").Build()).
Build()).
Build(),
cluster: builder.Cluster(metav1.NamespaceDefault, "cluster1").
WithTopology(
builder.ClusterTopology().
WithClass("class1").
WithVersion("v1.22.2").
WithMachineDeployment(
builder.MachineDeploymentTopology("forward/slash").
WithClass("aa").
Build()).
Build()).
Build(),
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
Expand Down Expand Up @@ -1540,6 +1600,66 @@ func TestMachinePoolTopologiesAreUniqueAndDefinedInClusterClass(t *testing.T) {
Build(),
wantErr: true,
},
{
name: "fail if MachinePoolTopology name is not a valid Kubernetes resource name",
clusterClass: builder.ClusterClass(metav1.NamespaceDefault, "class1").
WithInfrastructureClusterTemplate(
builder.InfrastructureClusterTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithControlPlaneTemplate(
builder.ControlPlane(metav1.NamespaceDefault, "cp1").Build()).
WithControlPlaneInfrastructureMachineTemplate(
builder.InfrastructureMachinePoolTemplate(metav1.NamespaceDefault, "cpinfra1").Build()).
WithWorkerMachinePoolClasses(
*builder.MachinePoolClass("aa").
WithInfrastructureTemplate(
builder.InfrastructureMachinePoolTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithBootstrapTemplate(
builder.BootstrapTemplate(metav1.NamespaceDefault, "bootstrap1").Build()).
Build()).
Build(),
cluster: builder.Cluster(metav1.NamespaceDefault, "cluster1").
WithTopology(
builder.ClusterTopology().
WithClass("class1").
WithVersion("v1.22.2").
WithMachinePool(
builder.MachinePoolTopology("under_score").
WithClass("aa").
Build()).
Build()).
Build(),
wantErr: true,
},
{
name: "fail if MachinePoolTopology name is not a valid Kubernetes label value",
clusterClass: builder.ClusterClass(metav1.NamespaceDefault, "class1").
WithInfrastructureClusterTemplate(
builder.InfrastructureClusterTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithControlPlaneTemplate(
builder.ControlPlane(metav1.NamespaceDefault, "cp1").Build()).
WithControlPlaneInfrastructureMachineTemplate(
builder.InfrastructureMachinePoolTemplate(metav1.NamespaceDefault, "cpinfra1").Build()).
WithWorkerMachinePoolClasses(
*builder.MachinePoolClass("aa").
WithInfrastructureTemplate(
builder.InfrastructureMachinePoolTemplate(metav1.NamespaceDefault, "infra1").Build()).
WithBootstrapTemplate(
builder.BootstrapTemplate(metav1.NamespaceDefault, "bootstrap1").Build()).
Build()).
Build(),
cluster: builder.Cluster(metav1.NamespaceDefault, "cluster1").
WithTopology(
builder.ClusterTopology().
WithClass("class1").
WithVersion("v1.22.2").
WithMachinePool(
builder.MachinePoolTopology("forward/slash").
WithClass("aa").
Build()).
Build()).
Build(),
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
Expand Down
84 changes: 84 additions & 0 deletions internal/webhooks/cluster_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1872,6 +1872,90 @@ func TestClusterTopologyValidation(t *testing.T) {
Build()).
Build(),
},
{
name: "should return error when MachineDeploymentTopology name is not a valid Kubernetes resource name",
expectErr: true,
in: builder.Cluster("fooboo", "cluster1").
WithTopology(builder.ClusterTopology().
WithClass("foo").
WithVersion("v1.19.1").
WithMachineDeployment(
builder.MachineDeploymentTopology("under_score").
WithClass("aa").
Build()).
Build()).
Build(),
},
{
name: "should return error when MachinePoolTopology name is not a valid Kubernetes resource name",
expectErr: true,
in: builder.Cluster("fooboo", "cluster1").
WithTopology(builder.ClusterTopology().
WithClass("foo").
WithVersion("v1.19.1").
WithMachinePool(
builder.MachinePoolTopology("under_score").
WithClass("aa").
Build()).
Build()).
Build(),
},
{
name: "should return error when MachineDeploymentTopology name is not a valid Kubernetes label value",
expectErr: true,
in: builder.Cluster("fooboo", "cluster1").
WithTopology(builder.ClusterTopology().
WithClass("foo").
WithVersion("v1.19.1").
WithMachineDeployment(
builder.MachineDeploymentTopology("forward/slash").
WithClass("aa").
Build()).
Build()).
Build(),
},
{
name: "should return error when MachinePoolTopology name is not a valid Kubernetes label value",
expectErr: true,
in: builder.Cluster("fooboo", "cluster1").
WithTopology(builder.ClusterTopology().
WithClass("foo").
WithVersion("v1.19.1").
WithMachinePool(
builder.MachinePoolTopology("forward/slash").
WithClass("aa").
Build()).
Build()).
Build(),
},
{
name: "should return error when MachineDeploymentTopology name exceeds 63 characters (the maximum length of a Kubernetes label value)",
expectErr: true,
in: builder.Cluster("fooboo", "cluster1").
WithTopology(builder.ClusterTopology().
WithClass("foo").
WithVersion("v1.19.1").
WithMachineDeployment(
builder.MachineDeploymentTopology("thisnameisreallymuchlongerthanthemaximumlengthofsixtythreecharacters").
WithClass("aa").
Build()).
Build()).
Build(),
},
{
name: "should return error when MachinePoolTopology name exceeds 63 characters (the maximum length of a Kubernetes label value)",
expectErr: true,
in: builder.Cluster("fooboo", "cluster1").
WithTopology(builder.ClusterTopology().
WithClass("foo").
WithVersion("v1.19.1").
WithMachinePool(
builder.MachinePoolTopology("thisnameisreallymuchlongerthanthemaximumlengthofsixtythreecharacters").
WithClass("aa").
Build()).
Build()).
Build(),
},
{
name: "should update",
expectErr: false,
Expand Down