How to Deploy a Prefect Agent with Plural

Deploying your Prefect agent using Plural is fast, easy, and requires almost zero knowledge of infrastructure as code (IaC), Kubernetes, or orchestration tools, such as Helm. In this tutorial, we’ll walk through the setup and deployment of a Prefect agent using the Plural console and a GCP environment.

Why integrate Plural and Prefect?

Plural and Prefect work side by side to provide the quickest and easiest deployment path for your Prefect agent so you can get up and running with some of the more advanced concepts that Prefect has to offer, namely deployments.

This means that you can go from a manual, locally run flow to scheduling deployments in minutes, while also avoiding the pitfalls of trying to set up your own infrastructure that a Prefect agent requires.

What are the benefits of deploying your Prefect Agent with Plural?

Plural takes care of orchestrating the deployment of the required cloud infrastructure for a Prefect agent as well as the deployment of the Prefect agent itself all in essentially three commands.

Plural also helps with understanding the health of your Prefect agent deployment through the use of the Plural console. The console can automate upgrades to your Prefect agent as those upgrades get pushed, as well as provide log access to the Prefect agent, or any other applications that you deploy with Plural.

What is a Prefect agent?

A Prefect agent is a lightweight polling service that pulls scheduled work from a work queue within a work pool and deploys the corresponding flow runs. A Prefect agent can live in a local environment, but most production implementations use deployments of the agent to a Kubernetes cluster or some sort of cloud infrastructure as a long-lived process.

What is Plural?

Plural is an open-source full-stack application delivery tool that can orchestrate the deployment of the infrastructure (cloud VPCs, Kubernetes clusters, etc.) and an application itself through the use of Helm. For Prefect users, it allows for a simplified and automated deployment of a Prefect agent — making using Prefect that much easier.

How Plural and Prefect work together

1. A user deploys the prefect agent through the Plural application via plural deploy (See full instructions below.)
2. The Prefect agent connects to your Prefect Cloud account and attaches itself to a work queue or a number of work queues. A Prefect agent lives as a kubernetes container and can then submit flows, pull flow data, and more, from your defined Prefect blocks.
3. Prefect Cloud will receive your deployment submission and schedule work through your Prefect agent.

Prerequisites:

• A Prefect Cloud account
• A GitHub or GitLab account

Install cloud provider prerequisites

Because Plural is a solution that deploys and manages infrastructure in a user’s cloud environment, it needs relatively high levels of access to your cloud environment.

As a result, you need to provide a service account to Plural so that it can authenticate against your cloud environment (AWS, GCP, or Azure.)

In this example, we’ll provide owner access to our resources in GCP, but you can tighten the permissions later to meet your security requirements.

Check out our documentation to quickly set up your cloud provider to integrate with Plural. Make sure to note the download location of the created service account(s)

Create a Plural account

You are ready to work with Plural and deploy the Prefect agent! Create a Plural account and then sign in. You should end up on a landing page similar to this:

Deploying the Prefect agent

Create a Plural Cloud shell

To deploy the Prefect agent, we will use a built-in tool in Plural’s web console called the cloud shell, though you can deploy Plural locally through your personal CLI by following the instructions here.

You will find the tool on the left-hand side of the web console. This tool creates a cloud environment with all the necessary Plural prerequisites installed, such as Terraform, Helm, kubectl, and Git.

1. Navigate to the Cloud Shell to create a GitHub or GitLab repository to store the state of the deployment.
   

2. Click Continue, and then select Cloud Shell, and click Continue once more.

3. Choose your cloud provider. Then choose the agent deployment region, provide the service account credentials you generated during the prerequisite stage, and click Continue.

4. Provide a memorable cluster name to be created for the deployment. Then, provide a unique bucket prefix and a subdomain for DNS creation. Note that deploying the agent on its own does not technically require a DNS entry, but Plural provides this for free, and it is required.

5. Click Continue, and then Create.

Deploying the Prefect agent

1. From the Plural Web Console, on the left-hand side, search for the “prefect-agent” and click “Continue.”

2. Enter a VPC name for the Prefect deployment to reside in.

3. Pass storage bucket information specific to your cloud environment if prompted (this is for the plural console which provides application health data and automated application upgrades.)

4.  Set a domain name for the Plural console.

5. Pass in your Prefect Cloud information:

You will also need to enter your Prefect Cloud API key (See here how to create an API key.) And, your Prefect Acount and Workspace ID.


6. Click Install and Plural will begin deploying the Plural console and the Prefect agent automatically.  You can follow the progress through the log output on the right hand side.  This normally takes 10-15 minutes.

7. Optional: After the deployment concludes, if you have specific work queues besides the default work queue that you would like the agent to poll, update the values file created from the plural build stage. The Plural cloud shell has a built-in code editor called vim that can be used to update this file:

8. Deploy the Prefect agent and add a commit message via the command plural deploy --commit "<COMMIT MESSAGE>" and wait for the deployment to complete.

9. Check your work queues in the Prefect Cloud console under your default work pool, and verify the agent is up and healthy.

10. Access your Plural Console to get real time status information on your Prefect agent deployment as well as the overall kubernetes cluster health. The plural console will also act as your hub for prefect-agent upgrades as updates are pushed.

More information on how the Plural console can benefit you can be found here.

Cleaning Up

1. From the Plural Cloud Shell run the plural destroy command, which will tear down all infra and Plural API refs, including the Plural console.

Next Steps

Congrats! You have deployed a Prefect agent, and the Plural state has been encrypted and stored in the generated GitHub repo for future use.

Now that you have completed the steps above, you can create your first Prefect deployment and set up your flow to run on a specified trigger or cron.

You can easily upgrade the Prefect agent and the underlying infrastructure at the click of a button using the included Plural console!

If you need some help, please feel free to reach out to Prefect via their Slack Channel or to our team over at Plural through our Discord Channel.

This post was co-written with Jimi Dodd-o a Senior Infrastructure Engineer at Prefect.