How to Get Runbooks From Kubernetes YAML With Plural

Make sure that you have installed Airflow and the Plural admin console. For instructions follow the getting started guide here.

From the root of your Plural installation folder, navigate to the application helm folder

cd <app-name>/helm/<app-name>/

Create a new .xml file called db-scaling.xml and paste in the following code:

<root gap='medium'>
<box pad='small' gap='medium' direction='row' align='center'>
<button label='Scale' action='scale' primary='true' headline='true' />
<box direction='row' align='center' gap='small'>
  <box gap='small' align='center'>
    <timeseries datasource="cpu" label="CPU Usage" />
    <text size='small'>You should set a reservation to 
      roughly correspond to 30% utilization</text>
  </box>
  <box gap='small' align='center'>
    <timeseries datasource="memory" label="Memory Usage" />
    <text size='small'>You should set a reservation to 
      roughly correspond to 60% utilization</text>
  </box>
</box>
<box gap='small'>
  <box gap='xsmall'>
    <input placeholder="250m" label='CPU Request' name='cpu'>
      <valueFrom 
        datasource="statefulset" 
        doc="kubernetes.raw" 
        path="spec.template.spec.containers[0].resources.requests.cpu" />
    </input>
    <input placeholder="1Gi" label='Memory Request' name='memory'>
      <valueFrom 
        datasource="statefulset" 
        doc="kubernetes.raw" 
        path="spec.template.spec.containers[0].resources.requests.memory" />
    </input>
  </box>
</box>
</box>
<box pad='small' gap='medium' direction='row' align='center'>
<box direction='row' width='70%' align='center'>
  <text size='small'>You can also add more replicas to provide failover in case of outages, or optionally remove them to save cost</text>
</box>
<box direction='row' gap='small' width='30%' align='center'>
  <input datatype='int' placeholder="1" label='Replicas' name='replicas'>
    <valueFrom 
      datasource="statefulset" 
      doc="kubernetes.raw" 
      path="spec.replicas" />
  </input>
</box>
</box>
<box width='100%' gap='small'>
<text size='small'>Be sure to scale your rabbitmq nodes within your nodes capacities, listed here:</text>
<table width='100%' datasource='nodes' path='nodes'>
  <tableColumn path='metadata.name' header='name' width='33%' />
  <tableColumn path='status.capacity.cpu' header='cpu' width='33%' />
  <tableColumn path='status.capacity.memory' header='memory' width='33%' />
</table>
</box>
</root>

This draws two boxes on a single row, each displaying a timeseries from the datasources cpu and memory respectively. Underneath, it draws another box with two input boxes, both of which is displaying a value from the datasource statefulset. It also draws a button with the scale action attached. Don't worry about these values right now, we'll come back to them below.

(For a full breakdown of the styling properties of the xml tags that have been configured by Plural, refer to the docs here.)

From the root of your Plural installation repo, navigate to the templates directory in your helm directory.

cd <app-name>/helm/<app-name>/templates

Create a runbook.yaml file

runbook.yaml is where you hydate your db-scaling.xml file with data sources. You can configure it a number of different ways, but in particular you'll want to:

• Define a display field -- this is where you link your xml template
• Define a datasources field -- this is where you put your datasources. Plural currently supports three data source types. You can view the full list here.
• Define an actions field -- this is the update to the application's helm chart that Plural will perform.

So for example:

apiVersion: platform.plural.sh/v1alpha1
kind: Runbook
metadata:
  name: db-scaling
  labels:
    platform.plural.sh/pinned: 'true'
{{ include "airflow.labels" . | indent 4 }}
spec:
  name: Postgres Scaling
  description: overview of how to accurately scale airflow's postgres instance
  alerts:
  - name: AirflowPostgresCPU
  - name: AirflowPostgresMEM
  display: |-
{{ .Files.Get "runbooks/db-scaling.xml" | indent 4 }}
  datasources:
  - name: cpu
    type: prometheus
    prometheus:
      format: cpu
      legend: $pod
      query: sum(rate(container_cpu_usage_seconds_total{namespace="{{ .Release.Namespace }}",pod=~"plural-airflow-[0-9]+"}[5m])) by (pod)
  - name: memory
    type: prometheus
    prometheus:
      format: memory
      legend: $pod
      query: sum(container_memory_working_set_bytes{namespace="{{ .Release.Namespace }}",pod=~"plural-airflow-[0-9]+"}) by (pod)
  - name: statefulset
    type: kubernetes
    kubernetes:
      resource: statefulset
      name: plural-airflow
  - name: nodes
    type: nodes
  actions:
  - name: scale
    action: config
    redirectTo: '/'
    configuration:
      updates:
      - path: 
        - airflow
        - postgres
        - resources
        - requests
        - cpu
        valueFrom: cpu
      - path:
        - airflow
        - postgres
        - resources
        - requests
        - memory
        valueFrom: memory
      - path:
        - airflow
        - postgres
        - replicas
        valueFrom: replicas

Build && Redeploy. From the root of your Plural installation folder, run:
plural build
git add .
git commit -m "Add runbook
git push
plural deploy

Navigate to the console subdomain, and click on the runbooks tab

Click on the runbook and you'll see a dashboard with memory + cpu metrics pulled from Prometheus, with the ability to scale those metrics.