Skip to content

Commit dd27307

Browse files
gnufiedgnufied
committed
Add a condition field for capturing status of resize on PVC
1 parent 9bb2192 commit dd27307

File tree

1 file changed

+81
-10
lines changed

1 file changed

+81
-10
lines changed

‎contributors/design-proposals/grow-volume-size.md‎

Lines changed: 81 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,12 @@ Enable users to increase size of PVs that their pods are using. The user will up
2525
| ----------------| :---------------: | :--------------------------:| :----------------------: |
2626
| EBS | Yes | Yes | Yes |
2727
| GCE PD | Yes | Yes | Yes |
28-
| Azure Disk | Yes | Yes | No |
28+
| GlusterFS | Yes | No | Yes |
2929
| Cinder | Yes | Yes | Yes |
3030
| Vsphere | Yes | Yes | No |
3131
| Ceph RBD | Yes | Yes | No |
3232
| Host Path | No | No | No |
33-
| GlusterFS | Yes | No | Yes |
33+
| Azure Disk | Yes | Yes | No |
3434
| Azure File | No | No | No |
3535
| Cephfs | No | No | No |
3636
| NFS | No | No | No |
@@ -63,13 +63,15 @@ For volume types that only require volume plugin based api call, this will be on
6363
A new controller called `volume_expand_controller` will listen for pvc size expansion requests and take action as needed. The steps performed in this
6464
new controller will be:
6565

66-
* Watch for pvc update requests and add pvc to controller's desired state of world if a increase in volume size was requested.
66+
* Watch for pvc update requests and add pvc to controller's desired state of world if a increase in volume size was requested. Once PVC is added to
67+
controller's desired state of world - `pvc.Status.Conditions` will be updated with `ResizeStarted: True`.
68+
* For unbound or pending PVCs - resize will trigger no action in `volume_expand_controller`.
6769
* A reconciler will read desired state of world and perform corresponding volume resize operation. If there is a resize operation in progress
6870
for same volume then resize request will be pending and retried once previous resize request has completed.
6971
* Controller resize in effect will be level based rather than edge based. If there are more than one pending resize request for same PVC then
7072
new resize requests for same PVC will replace older pending request.
7173
* Resize will be performed via volume plugin interface, executed inside a goroutine spawned by `operation_exectutor`.
72-
* A new plugin interface called `volume.Exander` will be added to volume plugin interface. The controller call to expand the PVC will look like:
74+
* A new plugin interface called `volume.Expander` will be added to volume plugin interface. The controller call to expand the PVC will look like:
7375

7476
```go
7577
func (og *operationGenerator) GenerateExpandVolumeFunc(
@@ -103,32 +105,64 @@ func (og *operationGenerator) GenerateExpandVolumeFunc(
103105
}
104106
```
105107

106-
* Once volume expand is successful, the volume will be marked as expanded and new size will be updated in `pv.spec.capacity`. Any errors will be
107-
reported as *events* on PVC object.
108+
* Once volume expand is successful, the volume will be marked as expanded and new size will be updated in `pv.spec.capacity`. Any errors will be reported as *events* on PVC object.
109+
* If resize failed in above step, in addition to events - `pvc.Status.Conditions` will be updated with `ResizeFailed: True`. Corresponding error will be added to condition field as well.
108110
* Depending on volume type next steps would be:
109111

110-
* If volume is of type that does not require file system resize, then `pvc.status.capacity` will be immediately updated to reflect new size. This would conclude the volume expand operation.
112+
* If volume is of type that does not require file system resize, then `pvc.status.capacity` will be immediately updated to reflect new size. This would conclude the volume expand operation. Also `pvc.Status.Conditions` will be updated with `Ready: True`.
111113
* If volume if of type that requires file system resize then a file system resize will be performed on kubelet. Read below for steps that will be performed for file system resize.
112114

113115
* If volume plugin is of type that can not do resizing of attached volumes (such as `Cinder`) then `ExpandVolumeDevice` can return error by checking for
114116
volume status with its own API (such as by making Openstack Cinder API call in this case). Controller will keep trying to resize the volume until it is
115117
successful.
116118

117119
* To consider cases of missed PVC update events, an additional loop will reconcile bound PVCs with PVs. This additional loop will loop through all PVCs
118-
and match `pvc.spec.capactiy` with `pv.spec.capacity` and add PVC in `volume_expand_controller`'s desired state of world if `pv.spec.capacity` is less
119-
than `pvc.spec.capacity`.
120+
and match `pvc.spec.resources.requests` with `pv.spec.capacity` and add PVC in `volume_expand_controller`'s desired state of world if `pv.spec.capacity` is less
121+
than `pvc.spec.resources.requests`.
120122

121123
* There will be additional checks in controller that grows PV size - to ensure that we do not make volume plugin API calls that can reduce size of PV.
122124

123125
### File system resize on kublet
124126

127+
A File system resize will be pending on PVC until a new pod that uses this volume is scheduled somewhere. While theoretically we *can* perform
128+
online file system resize if volume type and file system supports it - we are leaving it for next iteration of this feature.
129+
125130
* When calling `MountDevice` or `Setup` call of volume plugin, volume manager will in addition compare `pv.spec.capacity` and `pvc.status.capacity` and if `pv.spec.capacity` is greater
126131
than `pvc.status.spec.capacity` then volume manager will additionally resize the file system of volume.
127132
* The call to resize file system will be performed inside `operation_generator.GenerateMountVolumeFunc`. `VolumeToMount` struct will be enhanced to store PVC as well.
128133
* Any errors during file system resize will be added as *events* to Pod object and mount operation will be failed.
134+
* If there are any errors during file system resize `pvc.Status.Conditions` will be updated with `ResizeFailed: True`. Any errors will be added to
135+
`Conditions` field.
129136
* File System resize will not be performed on kubelet where volume being attached is ReadOnly. This is similar to pattern being used for performing formatting.
130-
* After file system resize is successful, `pvc.status.capacity` will be updated to match `pv.spec.capacity` and volume expand operation will be considered complete.
137+
* After file system resize is successful, `pvc.status.capacity` will be updated to match `pv.spec.capacity` and volume expand operation will be considered complete. Also `pvc.Status.Conditions` will be updated with `Ready: True`.
138+
139+
#### Reduce coupling between resize operation and file system type
140+
141+
A file system resize in general requires presence of tools such as `resize2fs` or `xfs_growfs` on the host where kubelet is running. There is a concern
142+
that open coding call to different resize tools direclty in Kubernetes will result in coupling between file system and resize operation. To solve this problem
143+
we have considered following options:
144+
145+
1. Write a library that abstracts away various file system operations, such as - resizing, formatting etc.
146+
147+
Pros:
148+
* Relatively well known pattern
149+
150+
Cons:
151+
* Depending on version with which Kubernetes is compiled with, we are still tied to which file systems are supported in which version
152+
of kubernetes.
153+
2. Ship a wrapper shell script that encapsulates various file system operations and as long as the shell script supports particular file system
154+
the resize operation is supported.
155+
Pros:
156+
* Kubernetes Admin can easily replace default shell script with her own version and thereby adding support for more file system types.
157+
158+
Cons:
159+
* I don't know if there is a pattern that exists in kube today for shipping shell scripts that are called out from code in Kubernetes. Flex is
160+
different because, none of the flex scripts are shipped with Kuberntes.
161+
3. Ship resizing tools in a container.
162+
131163

164+
Of all options - #3 is our best bet but we are not quite there yet. Hence, I would like to propose that we ship with support for
165+
most common file systems in curent release and we revisit this coupling and solve it in next release.
132166

133167
## API and UI Design
134168

@@ -173,6 +207,43 @@ spec:
173207
174208
`pvc.spec.resources.requests.storage` field of pvc object will become mutable after this change.
175209

210+
In addition to that PVC's status will have a `Conditions []PvcCondition` - which will be used
211+
to communicate the status of PVC to the user.
212+
213+
So the `PersistentVolumeClaimStatus` will become:
214+
215+
```go
216+
type PersistentVolumeClaimStatus struct {
217+
Phase PersistentVolumeClaimPhase
218+
AccessModes []PersistentVolumeAccessMode
219+
Capacity ResourceList
220+
// New Field added as part of this Change
221+
Conditions []PVCCondition
222+
}
223+
224+
// new API type added
225+
type PVCCondition struct {
226+
Type PVCConditionType
227+
Status ConditionStatus
228+
LastProbeTime metav1.Time
229+
LastTransitionTime metav1.Time
230+
Reason string
231+
Message string
232+
}
233+
234+
// new API type
235+
type PVCConditionType string
236+
237+
// new Constants
238+
const (
239+
PVCReady PVCConditionType = "Ready"
240+
PVCResizeStarted PVCConditionType = "ResizeStarted"
241+
PVCResizeFailed PVCResizeFailed = "ResizeFailed"
242+
)
243+
```
244+
245+
246+
176247
### Other API changes
177248

178249
This proposal relies on ability to update PVC status from kubelet. While updating PVC's status

0 commit comments

Comments
 (0)