How to Get Runbooks From Kubernetes YAML With Plural
At Plural, we provide at Plural is programmatic runbooks integrated with your deployment pipeline that make common manual DevOps tasks a breeze.
At Plural, we're making it easy to deploy open-source applications like Prometheus, Grafana, and RabbitMQ, on Kubernetes.
One of the features our open-source product provides is programmatic runbooks that directly integrate with your deployment pipeline making common manual DevOps tasks like scaling CPU and memory a breeze.
In this blog post, we'll walk you through how you can write your own runbook from scratch.
Writing Your Own Plural Runbook
Plural Runbooks are meant to be installed alongside your open source applications and serve as interactive tutorials for how to perform common maintenance tasks. In this tutorial we'll create a runbook that presents a nice UI for scaling Airflow's Postgres instance.
-
Make sure that you have installed Airflow and the Plural admin console. For instructions follow the getting started guide here.
-
From the root of your Plural installation folder, navigate to the application helm folder
cd <app-name>/helm/<app-name>/
-
Create a new
.xml
file calleddb-scaling.xml
and paste in the following code:<root gap='medium'> <box pad='small' gap='medium' direction='row' align='center'> <button label='Scale' action='scale' primary='true' headline='true' /> <box direction='row' align='center' gap='small'> <box gap='small' align='center'> <timeseries datasource="cpu" label="CPU Usage" /> <text size='small'>You should set a reservation to roughly correspond to 30% utilization</text> </box> <box gap='small' align='center'> <timeseries datasource="memory" label="Memory Usage" /> <text size='small'>You should set a reservation to roughly correspond to 60% utilization</text> </box> </box> <box gap='small'> <box gap='xsmall'> <input placeholder="250m" label='CPU Request' name='cpu'> <valueFrom datasource="statefulset" doc="kubernetes.raw" path="spec.template.spec.containers[0].resources.requests.cpu" /> </input> <input placeholder="1Gi" label='Memory Request' name='memory'> <valueFrom datasource="statefulset" doc="kubernetes.raw" path="spec.template.spec.containers[0].resources.requests.memory" /> </input> </box> </box> </box> <box pad='small' gap='medium' direction='row' align='center'> <box direction='row' width='70%' align='center'> <text size='small'>You can also add more replicas to provide failover in case of outages, or optionally remove them to save cost</text> </box> <box direction='row' gap='small' width='30%' align='center'> <input datatype='int' placeholder="1" label='Replicas' name='replicas'> <valueFrom datasource="statefulset" doc="kubernetes.raw" path="spec.replicas" /> </input> </box> </box> <box width='100%' gap='small'> <text size='small'>Be sure to scale your rabbitmq nodes within your nodes capacities, listed here:</text> <table width='100%' datasource='nodes' path='nodes'> <tableColumn path='metadata.name' header='name' width='33%' /> <tableColumn path='status.capacity.cpu' header='cpu' width='33%' /> <tableColumn path='status.capacity.memory' header='memory' width='33%' /> </table> </box> </root>
This draws two boxes on a single row, each displaying a timeseries from the datasources cpu and memory respectively. Underneath, it draws another box with two input boxes, both of which is displaying a value from the datasource
statefulset
. It also draws a button with thescale
action attached. Don't worry about these values right now, we'll come back to them below.(For a full breakdown of the styling properties of the xml tags that have been configured by Plural, refer to the docs here.)
-
From the root of your Plural installation repo, navigate to the templates directory in your helm directory.
cd <app-name>/helm/<app-name>/templates
-
Create a
runbook.yaml
filerunbook.yaml
is where you hydate yourdb-scaling.xml
file with data sources. You can configure it a number of different ways, but in particular you'll want to:- Define a
display
field -- this is where you link yourxml
template - Define a
datasources
field -- this is where you put your datasources. Plural currently supports three data source types. You can view the full list here. - Define an
actions
field -- this is the update to the application's helm chart that Plural will perform.
So for example:
apiVersion: platform.plural.sh/v1alpha1 kind: Runbook metadata: name: db-scaling labels: platform.plural.sh/pinned: 'true' {{ include "airflow.labels" . | indent 4 }} spec: name: Postgres Scaling description: overview of how to accurately scale airflow's postgres instance alerts: - name: AirflowPostgresCPU - name: AirflowPostgresMEM display: |- {{ .Files.Get "runbooks/db-scaling.xml" | indent 4 }} datasources: - name: cpu type: prometheus prometheus: format: cpu legend: $pod query: sum(rate(container_cpu_usage_seconds_total{namespace="{{ .Release.Namespace }}",pod=~"plural-airflow-[0-9]+"}[5m])) by (pod) - name: memory type: prometheus prometheus: format: memory legend: $pod query: sum(container_memory_working_set_bytes{namespace="{{ .Release.Namespace }}",pod=~"plural-airflow-[0-9]+"}) by (pod) - name: statefulset type: kubernetes kubernetes: resource: statefulset name: plural-airflow - name: nodes type: nodes actions: - name: scale action: config redirectTo: '/' configuration: updates: - path: - airflow - postgres - resources - requests - cpu valueFrom: cpu - path: - airflow - postgres - resources - requests - memory valueFrom: memory - path: - airflow - postgres - replicas valueFrom: replicas
- Define a
-
Build && Redeploy. From the root of your Plural installation folder, run:
plural build
git add .
git commit -m "Add runbook
git push
plural deploy
-
Navigate to the console subdomain, and click on the runbooks tab
Click on the runbook and you'll see a dashboard with memory + cpu metrics pulled from Prometheus, with the ability to scale those metrics.
Next Steps With Plural
Plural comes with built-in runbooks for popular use cases, and you can use them alongside your Plural-deployed open source applications as is.
For more about Plural, our full docs are here and you can check out our code repository at https://github.com/pluralsh/plural.
If you run into any problems or have suggestions for what else you’d like to use Plural for, please let us know in our Discord.
Newsletter
Be the first to know when we drop something new.