Skip to content

Commit 9bb2192

Browse files
gnufiedgnufied
committed
Add more details and real world use cases for resize
1 parent 91b4102 commit 9bb2192

File tree

1 file changed

+84
-61
lines changed

1 file changed

+84
-61
lines changed

‎contributors/design-proposals/grow-volume-size.md‎

Lines changed: 84 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,13 @@ Enable users to increase size of PVs that their pods are using. The user will up
1515
* As a user I am running Mysql on a 100GB volume - but I am running out of space, I should be able to increase size of volume mysql is using without losing all my data. (*online and with data*)
1616
* As a user I created a PVC requesting 2GB space. I am yet to start a pod with this PVC but I realize that I probably need more space. Without having to create a new PVC, I should be able to request more size with same PVC. (*offline and no data on disk*)
1717
* As a user I was running a rails application with 5GB of assets PVC. I have taken my application offline for maintenance but I would like to grow asset PVC to 10GB in size. (*offline but with data*)
18+
* As a user I am running an application on glusterfs. I should be able to resize the gluster volume without losing data or mount point. (*online and with data and without taking pod offline*)
19+
* In the logging project we run on dedicated clusters, we start out with 187Gi PVs for each of the elastic search pods. However, the amount of logs being produced can vary greatly from one cluster to another and its not uncommon that these volumes fill and we need to grow them.
1820

1921
## Volume Plugin Matrix
2022

2123

22-
| Volume Plugin | Supports Resize | Requires File system Resize | Supported in 1.7 Release |
24+
| Volume Plugin | Supports Resize | Requires File system Resize | Supported in 1.8 Release |
2325
| ----------------| :---------------: | :--------------------------:| :----------------------: |
2426
| EBS | Yes | Yes | Yes |
2527
| GCE PD | Yes | Yes | Yes |
@@ -36,34 +38,97 @@ Enable users to increase size of PVs that their pods are using. The user will up
3638

3739
## Implementation Design
3840

39-
For volume type that supports growing the PV size, this will be a two step operation:
41+
For volume type that requires both file system expansion and a volume plugin based modification, growing persistent volumes will be two
42+
step process.
4043

41-
* A controller in master-controller will listen for PVC events and perform corresponding cloudprovider operation. If successful - controller will store new device size in PV. Some cloudproviders (such as cinder) - do not allow resizing of attached volumes. In such cases - it is upto volume plugin maintainer to decide appropriate behaviour. Volume Plugin maintainer can choose to ignore resize request if disk is attached to a pod (and add appropriate error events to PVC object). Resize request will keep failing until user corrects the error. User can take necessary action in such cases (such as scale down the pod) which will allow resize to proceed normally.
4244

43-
In case where volume type requires no file system resize, both PV & PVC objects will be updated accordingly and `status.capacity` of both objects will reflect new size.
44-
For volume plugins that require file system resize - an additional annotation called `volume.alpha.kubernetes.io/fs-resize-pending` will be added to PV to communicate
45-
to the Kubelet that File system must be resized when a new pod is started using the PV.
45+
For volume types that only require volume plugin based api call, this will be one step process.
4646

47-
* In case volume plugin doesn’t support resize feature. The resize API request will be rejected and PVC object will not be saved. This check will be performed via an admission controller plugin.
47+
### Prerequisite
48+
49+
* `pvc.spec.resources.requests.storage` field of pvc object will become mutable after this change.
50+
* #sig-api-machinery has agreed to allow pvc's status update from kubelet as long as pvc and node relationship
51+
can be validated by node authorizer.
52+
* This feature will be protected by an alpha feature gate.
53+
54+
### Admission Control and Validations
4855

56+
* Resource quota code has to be updated to take into account PVC expand feature.
57+
* In case volume plugin doesn’t support resize feature. The resize API request will be rejected and PVC object will not be saved. This check will be performed via an admission controller plugin.
4958
* In case requested size is smaller than current size of PVC. A validation will be used to reject the API request. (This could be moved to admission controller plugin too.)
5059

51-
* There will be additional checks in controller that grows PV size - to ensure that we do not make cloudprovider API calls that can reduce size of PV.
5260

53-
* To consider cases of missed PVC update events, an additional loop will reconcile bound PVCs with PVs.
61+
### Controller Manager resize
62+
63+
A new controller called `volume_expand_controller` will listen for pvc size expansion requests and take action as needed. The steps performed in this
64+
new controller will be:
65+
66+
* Watch for pvc update requests and add pvc to controller's desired state of world if a increase in volume size was requested.
67+
* A reconciler will read desired state of world and perform corresponding volume resize operation. If there is a resize operation in progress
68+
for same volume then resize request will be pending and retried once previous resize request has completed.
69+
* Controller resize in effect will be level based rather than edge based. If there are more than one pending resize request for same PVC then
70+
new resize requests for same PVC will replace older pending request.
71+
* Resize will be performed via volume plugin interface, executed inside a goroutine spawned by `operation_exectutor`.
72+
* A new plugin interface called `volume.Exander` will be added to volume plugin interface. The controller call to expand the PVC will look like:
73+
74+
```go
75+
func (og *operationGenerator) GenerateExpandVolumeFunc(
76+
pvcWithResizeRequest *expandcache.PvcWithResizeRequest,
77+
dsow expandcache.DesiredStateOfWorld) (func() error, error) {
78+
79+
volumePlugin, err := og.volumePluginMgr.FindExpandablePluginBySpec(pvcWithResizeRequest.VolumeSpec)
80+
81+
if err != nil {
82+
return nil, fmt.Errorf("Error finding plugin for expanding volume: %q with error %v", pvcWithResizeRequest.UniquePvcKey(), err)
83+
}
84+
85+
expanderPlugin, err := volumePlugin.NewExpander()
86+
87+
if err != nil {
88+
return nil, fmt.Errorf("Error creating expander plugin for volume %q with error %v", pvcWithResizeRequest.UniquePvcKey(), err)
89+
}
90+
91+
expandFunc := func() error {
92+
expandErr := expanderPlugin.ExpandVolumeDevice(pvcWithResizeRequest.VolumeSpec, pvcWithResizeRequest.ExpectedSize, pvcWithResizeRequest.CurrentSize)
93+
94+
if expandErr != nil {
95+
glog.Errorf("Error expanding volume through cloudprovider : %v", expandErr)
96+
return expandErr
97+
}
98+
dsow.MarkAsResized(pvcWithResizeRequest)
99+
100+
return nil
101+
}
102+
return expandFunc, nil
103+
}
104+
```
105+
106+
* Once volume expand is successful, the volume will be marked as expanded and new size will be updated in `pv.spec.capacity`. Any errors will be
107+
reported as *events* on PVC object.
108+
* Depending on volume type next steps would be:
109+
110+
* If volume is of type that does not require file system resize, then `pvc.status.capacity` will be immediately updated to reflect new size. This would conclude the volume expand operation.
111+
* If volume if of type that requires file system resize then a file system resize will be performed on kubelet. Read below for steps that will be performed for file system resize.
112+
113+
* If volume plugin is of type that can not do resizing of attached volumes (such as `Cinder`) then `ExpandVolumeDevice` can return error by checking for
114+
volume status with its own API (such as by making Openstack Cinder API call in this case). Controller will keep trying to resize the volume until it is
115+
successful.
54116

55-
* Resource Quota code in admission controller has to be updated to consider PVC updates.
117+
* To consider cases of missed PVC update events, an additional loop will reconcile bound PVCs with PVs. This additional loop will loop through all PVCs
118+
and match `pvc.spec.capactiy` with `pv.spec.capacity` and add PVC in `volume_expand_controller`'s desired state of world if `pv.spec.capacity` is less
119+
than `pvc.spec.capacity`.
56120

57-
* The resize of file system will be performed on kubelet. If there is a running pod - no operation will be performed. Only when a new pod is started using same PVC - then kubelet will match device size and size of pv and attempt a resize of file system. resizing filesystem will be a volume plugin function. It is upto volume plugin maintainer to correctly implement this. In following cases no resize will be necessary and hence volume plugin can return success without actually doing anything.
121+
* There will be additional checks in controller that grows PV size - to ensure that we do not make volume plugin API calls that can reduce size of PV.
58122

59-
* If disk being attached to the pod is unformatted. In which case since kubelet formats the disk, no resize is necessary.
60-
* If PVC being attached to pod is of volume type that requires no file system level resize. Such as glusterfs.
123+
### File system resize on kublet
61124

62-
Once file system resize is successful - kubelet will update `pv.spec.status.capacity` and `pvc.spec.status.capacity`field to reflect updated size. Kubelet will also
63-
update `storageCapacityCondition` and remove the `volume.alpha.kubernetes.io/fs-resize-pending` annotation.
125+
* When calling `MountDevice` or `Setup` call of volume plugin, volume manager will in addition compare `pv.spec.capacity` and `pvc.status.capacity` and if `pv.spec.capacity` is greater
126+
than `pvc.status.spec.capacity` then volume manager will additionally resize the file system of volume.
127+
* The call to resize file system will be performed inside `operation_generator.GenerateMountVolumeFunc`. `VolumeToMount` struct will be enhanced to store PVC as well.
128+
* Any errors during file system resize will be added as *events* to Pod object and mount operation will be failed.
129+
* File System resize will not be performed on kubelet where volume being attached is ReadOnly. This is similar to pattern being used for performing formatting.
130+
* After file system resize is successful, `pvc.status.capacity` will be updated to match `pv.spec.capacity` and volume expand operation will be considered complete.
64131

65-
* File System resize will not be performed on kubelet where volume being attached is ReadOnly.
66-
* Once disk has been provisioned with new size, it will be mounted and used in a pod as usual.
67132

68133
## API and UI Design
69134

@@ -104,56 +169,14 @@ spec:
104169
105170
## API Changes
106171
107-
### PV API Change
108-
109-
Two new fields will be added to `PersistentVolumeStatus` object. One is `capacity` and another is `storageCapacityCondition`.
110-
111-
`storageCapacityCondition` field could be just annotation in Alpha. This field will become true if `spec.capacity.storage` and `status.capacity.storage` match their values.
112-
An additional `volume.alpha.kubernetes.io/fs-resize-pending` annotation will be added by controller to indicate that - `PersistentVolume` needs file system resize.
113-
114-
115-
```go
116-
type ResourceList map[ResourceName]resource.Quantity
117-
118-
type PersistentVolumeStatus struct {
119-
Capacity ResourceList
120-
StorageCapacityCondition bool
121-
}
122-
```
123-
124-
For example - YAML representation of a PV undergoing resize will become:
125-
126-
```yaml
127-
apiVersion: v1
128-
kind: PersistentVolume
129-
metadata:
130-
name: pv0003
131-
spec:
132-
capacity:
133-
# size requested
134-
storage: 10Gi
135-
accessModes:
136-
- ReadWriteOnce
137-
persistentVolumeReclaimPolicy: Recycle
138-
status:
139-
capacity:
140-
# actual size
141-
storage: 5Gi
142-
storageCapacityCondition: false
143-
```
144-
145-
146172
### PVC API Change
147173
148174
`pvc.spec.resources.requests.storage` field of pvc object will become mutable after this change.
149175

150-
Similar to PV, PVC API object will have `storageCapacityCondition` field:
151-
`storageCapacityCondition` field could be just annotation in Alpha.
152-
153176
### Other API changes
154177

155-
This proposal relies on ability to update PV & PVC objects from kubelet. Kubelet policy has to be relaxed
156-
to enabled that - https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go#L204-L247
178+
This proposal relies on ability to update PVC status from kubelet. While updating PVC's status
179+
a PATCH request must be made from kubelet to update the status.
157180

158181
Also - an Admin can directly edit the PV and specify new size but controller will not perform
159182
any automatic resize of underlying volume or file system in such cases.

0 commit comments

Comments
 (0)