Update document about error handling #5462
Conversation
k8s-ci-robot
added
cncf-cla: yes
k8s-ci-robot
added
the
sig/storage
|
/assign @msau42 @jsafrane @xing-yang |
k8s-ci-robot
added
the
size/M
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: gnufied The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
| In general Kubernetes sidecars classify all CSI errors in three different classes. Namely: | ||
|
|
||
| - Non-final errors (such as `DeadlineExceeded`), which indicate a transient error, which may be because of timeout or some other temporary failure. |
There was a problem hiding this comment.
Would add to non-final errors that "CO is unsure if volume was modified"
Because CSI Operation CAN time out despite volume modification occurring in storage provider. E.g. storage providers where modifications may take a while
There was a problem hiding this comment.
Kinda added extra wording. But not exactly what you wrote above. Please do check.
|
|
||
| #### Handling of infeasible errors | ||
|
|
||
| If volume modification to a VAC is failing with a final and infeasible error, then users can either set VAC to previously specified value in `status.currentVolumeAttributesClass` or set to `nil` if no VAC was specified. In both the cases, external-resizer will stop trying to reconcile the volume modification. |
There was a problem hiding this comment.
If volume modification to a VAC is failing with a final and infeasible error
I thought we ONLY cancel modification on infeasible err.
This is to prevent partial modification on final errs like InternalErr, which could lead to half-modified volumes for drivers.
There was a problem hiding this comment.
I was also thinking we ONLY cancel modification(rollback) on Infeasible err, kubernetes-csi/external-resizer#487 is based on this assumption.
There was a problem hiding this comment.
I think he's just stating that infeasible errors are a subset of final errors. All infeasible errors are final, but not all final errors are infeasible. This should probably say:
"failing with an infeasible error (but not other final errors),"
There was a problem hiding this comment.
Yes I meant and to do some heavy lifting here. Since infeasible are already final, both conditions must be true. I will update the wording.
|
|
||
| #### Handling of infeasible errors | ||
|
|
||
| If volume modification to a VAC is failing with a final and infeasible error, then users can either set VAC to previously specified value in `status.currentVolumeAttributesClass` or set to `nil` if no VAC was specified. In both the cases, external-resizer will stop trying to reconcile the volume modification. |
There was a problem hiding this comment.
I think he's just stating that infeasible errors are a subset of final errors. All infeasible errors are final, but not all final errors are infeasible. This should probably say:
"failing with an infeasible error (but not other final errors),"
k8s-ci-robot
added
the
lgtm
|
|
||
| Please note if PVC already had a `currentVolumeAttributesClass` in its status, then setting VAC to `nil` is not allowed. | ||
|
|
||
| It is possible that if there were one or more partial volume modifications that happened before on the volume, they will not be undone when this happens because for infeasible errors no `ControllerModifyVolume` will be called when user resets the VAC. This mechanism exists only to prevent perpetual call to `ControllerModifyVolume` for volume modifications which are never going to succeed. Storage providers and users are recommended to roll forward to different VAC, if desired behaviour is resetting the VAC to some pre-specified value for all `mutable_parameters`. |
There was a problem hiding this comment.
As a developer of CSI driver, and a cluster admin of our infra, I still cannot accept this.
they will not be undone
This means, when I specify my volume to have 2000 IOPS, and PVC.status tells me the reconcile finishes, but my volume may actually have only 1000 IOPS. And I can never observe the abnormal from Kubernetes API, until something more serious goes wrong:
- If the performance is higher than expected, it will incur extra cost
- If the performance is lower than expected, it can result in unexpected latency to workload, even catastrophic system failure
- If we add topology integration to VAC, it also means then PV nodeAffinity can be out-of-sync, which will cause Pod pending or stuck due to scheduled to wrong node.
- It is also subject to potential quota abuse
This mechanism exists only to prevent perpetual call
This does not make sense. After VAC is rolled back, if the volume is already at the desired state, SP should just return OK and do nothing. There is no reason the call will be perpetual. If the volume is actually partially modified, and cannot be rolled back by SP, it is better to let user notice this, rather than just hide it.
We never end the reconcile process with an failed gRPC call. e.g.
- We only delete VolumeAttachment if ControllerUnpublishVolume returns OK.
- We only clear PVC.Status.AllocatedResourceStatuses if ControllerExpandVolume returns OK.
So we should do the same, only clear PVC.Status.ModifyVolumeStatus if ControllerModifyVolume returns OK, and never cancel modification.
Storage providers and users are recommended to roll forward to different VAC, if desired behaviour is resetting the VAC to some pre-specified value for all
mutable_parameters.
In Kubernetes, spec specifies the desired state, not action. Which ever state the user specifies, we should try to bring the underlying system to the specified state. It would be ridiculous if two VAC specifies the same state, but only one of them will work.
k8s-ci-robot
removed
the
lgtm
|
New changes are detected. LGTM label has been removed. |
This PR documents error handling in external-resizer and how each type of error is handled.