Resizing a PersistentVolume Claimed by a StatefulSet with the Plural Operator
Suppose you're running a MySQL deployment on Kubernetes and the space that you allocated initially is not enough – you would like to increase it from 50GB
to 100GB
. How would you go about doing this?
Well, recall that with Kubernetes, you deploy stateless applications with Deployment
and stateful applications (like MySQL) with StatefulSet
. StatefulSet
manages pods that are backed by a Kubernetes storage object, for example PersistentVolume
.
With Kubernetes v1.11, you can now resize an existing PersistentVolume
simply by editing the PersistentVolumeClaim
(PVC) object. However, there is still no way to automatically resize volumes that are managed by a StatefulSet
.
Here at Plural, our goal is to make it easy to deploy open source applications like Airflow, Grafana, and RabbitMQ on Kubernetes.
The Plural operator automates some manual tasks that are involved in the setup of the more complicated open source applications.
For example, in order to resize a PersistentVolume
that has been claimed by a StatefulSet
object, the Plural Operator controller implements the following steps:
-
Delete the
StatefulSet
object without deleting the underlying pods.log.Info("deleting statefulset while orphaning pods", "statefulset", statefulset.Name) if err := r.Delete(ctx, &statefulset, client.PropagationPolicy(metav1.DeletePropagationOrphan)); err != nil { log.Error(err, "failed to delete", "statefulset", statefulset.Name) return ctrl.Result{}, err }
-
Set
allowVolumeExpansion: true
on theStorageClass
associated with your PVC.// find the storageClass associated with your PVC var storageClass storagev1.StorageClass if claim.Spec.StorageClassName == nil { classes := &storagev1.StorageClassList{} if err := r.List(ctx, classes); err != nil { log.Error(err, "could not list storage classes") return ctrl.Result{}, err } found := false for _, class := range classes.Items { fmt.Printf("%+v", class) if _, ok := class.Annotations["storageclass.kubernetes.io/is-default-class"]; ok { storageClass = class found = true break } } if !found { err := fmt.Errorf("Could not find default storage class") log.Error(err, "could not find default storage class") return ctrl.Result{}, err } } else { if err := r.Get(ctx, types.NamespacedName{Name: *claim.Spec.StorageClassName}, &storageClass); err != nil { log.Error(err, "failed to get storageClass", "storageclass", *claim.Spec.StorageClassName) return ctrl.Result{}, err } } // set AllowVolumeExpansion on the PVC if storageClass.AllowVolumeExpansion == nil || !*storageClass.AllowVolumeExpansion { storageClass.AllowVolumeExpansion = resources.BoolPtr(true) if err := r.Update(ctx, &storageClass); err != nil { log.Error(err, "Failed to enable volume expansion for storage class", "storageclass", storageClass.Name) return ctrl.Result{}, err } }
-
Modify the
PVC
object with the desired storagefor i := 0; i < replicas; i++ { claimName := fmt.Sprintf("%s-%s-%d", claim.Name, statefulset.Name, i) var pvc corev1.PersistentVolumeClaim nsn := types.NamespacedName{ Namespace: resize.Namespace, Name: claimName, } if err := r.Get(ctx, nsn, &pvc); err != nil { log.Error(err, "failed to get pvc", "pvc", pvc.Name) return ctrl.Result{}, err } pvc.Spec.Resources.Requests["storage"] = quant if err := r.Update(ctx, &pvc); err != nil { log.Error(err, "failed to resize statefulset pvc", "pvc", pvc.Name) return ctrl.Result{}, err } }
-
Recreate the
StatefulSet
with the new storage request// create a new set of VolumeClaims claims := statefulset.Spec.VolumeClaimTemplates newClaims := make([]corev1.PersistentVolumeClaim, len(claims)) var claim corev1.PersistentVolumeClaim for i, cl := range claims { if cl.Name == resize.Spec.PersistentVolume { if quant.Cmp(cl.Spec.Resources.Requests["storage"]) == 0 { log.Info("No change needed for storage") return r.cleanup(ctx, &resize) } cl.Spec.Resources.Requests["storage"] = quant claim = cl } newClaims[i] = cl } // create a new StatefulSet and set the volumeClaimTemplates with the new claims newStatefulSet := &appsv1.StatefulSet{} newStatefulSet.Spec = statefulset.Spec newStatefulSet.Spec.VolumeClaimTemplates = newClaims newStatefulSet.Name = statefulset.Name newStatefulSet.Namespace = statefulset.Namespace newStatefulSet.Labels = statefulset.Labels newStatefulSet.Annotations = statefulset.Annotations if newStatefulSet.Spec.Template.Annotations == nil { newStatefulSet.Spec.Template.Annotations = map[string]string{} } newStatefulSet.Spec.Template.Annotations["platform.plural.sh/rotate"] = time.Now().String() if err := r.Create(ctx, newStatefulSet); err != nil { log.Error(err, "failed to recreate statefulset", "statefulset", newStatefulSet.Name) return ctrl.Result{}, err }
Next Steps With Plural
That was a quick overview of how storage scaling is automatically handled for your Plural-installed applications. Plural offers a number of other goodies, in particular the Admin Console which serves as a central control panel.
For more about Plural, our full docs are at docs.plural.sh.
If you run into any problems or have suggestions for what else you’d like to use Plural for, please let us know in our Discord.
Newsletter
Be the first to know when we drop something new.