How to Deploy Dagster on Kubernetes

Dagster is an open-source data orchestration platform designed to develop, produce, and observe data assets. The recently created open-source project targets the same problem that Airflow solves but features a modern cloud-native design.

What separates Dagster from Airflow is Dagster’s support of a declarative, asset-based approach to orchestration. With Dagster, you can focus on delivering critical data assets and not on the tasks of pipelines.

Currently, if you were to deploy Dagster yourself on Kubernetes, you would need to do the following:

• Manually provision a Kubernetes cluster
• Manually install Dagster’s Helm chart
• Manage applications, which includes handling upgrades, scaling, and reliability issues

While this is doable, managing production-stage operations and monitoring requires frequent manual intervention. Additionally, this process varies across different cloud providers and platforms.

This article will go over setting up a fresh Kubernetes cluster and installing Dagster onto that cluster using Plural, a free, open-source Kubernetes DevOps platform that allows you to deploy production-ready Kubernetes clusters and open-source applications with little to no management experience necessary.

Let’s get started.

Prerequisites

Before installing Dagster on a Kubernetes cluster, you’ll need to create an account with Plural. While it may feel weird to create an account for an open-source tool it allows us to provide you with the following benefits:

• A cloud shell that bypasses the need for a local terminal
• A single source of truth for all your open-source installations and dependencies
• Auth tracking to ensure a user has permission to edit and configure the Plural installation
• An Audit Log viewer
• …and much more.

To create an account with Plural head over to app.plural.sh and follow the on-screen instructions. If you prefer to use Plural in your command line, follow our CLI quickstart guide which dives into installing and managing applications with the CLI.

Note: For this tutorial, we’ll use our own GCP cloud credentials since we already have a service account set up with GCP. Check out our documentation to quickly set up your own cloud provider to integrate with our platform (and make sure to note the download location of the created service account.)

How to create a Kubernetes Cluster

1. After signing up for Plural, you will enter our onboarding experience.

2. Click on use your own Cloud (We recommend users use our cloud shell experience.)

3. Create a GitHub or GitLab repository to store the state of the deployment. Plural manages all cluster configurations via Git and will provision a GitHub repository on your behalf. This repository is set up using scoped deploy keys to store the state of your workspace, and no OAuth credentials are persisted.

4. Choose your cloud provider. Plural is a solution that deploys and manages infrastructure in a user’s cloud environment, it needs relatively high levels of access to your cloud environment. As a result, you need to provide a service account to Plural so it can authenticate against your cloud environment.

5. Enter a unique cluster name to be created for the deployment. Then, provide a unique bucket prefix and a subdomain for DNS creation.

6. Review that the information you entered is correct, and if so click create. Note: This step can take a few minutes.

Plural will spin up a cloud shell environment for you, and now you’ll be prompted to select the applications you want to install on a fresh Kubernetes cluster.

How to Install Dagster on a Kubernetes Cluster

For this demo, we will install Dagster and the Plural console, a web-based dashboard that allows you to manage all your Plural applications and clusters in one place.

2. Set up some basic workspace configuration to ensure we name the cluster to add. You’ll first enter a VPC name for the Dagster deployment to reside in, ensuring Plural has a clean environment to deploy into and minimizes disruption to existing systems.

3. If you installed the Plural Console earlier you’ll be prompted to enter the storage bucket information specific to your cloud environment. The Plural Console provides application health data and automated application upgrades. Plural uses a tool called Renovate to automate the updating of application version images. Renovate creates pull requests against our plural-artifacts GitHub repository to perform these updates regularly. On our end, we test out the changes individually and once our team confirms the new version is stable we merge the PR and the update will be available for all Plural installations to pull down when they are ready

4. Next, If you earlier chose to install the Plural Console alongside your Dagster installation you’ll be prompted to configure your Plural Console environment.

5. Enter a hostname for you to access your airbyte deployment as well as a private DNS name to securely access the Dagster API. Note: It's recommended to name the hostname after the application.

6. Click Install and Plural will begin deploying the Plural console and Dagster automatically. Plural will run ‘plural build’ and ‘plural deploy’ on your behalf. You can follow the progress through the log output on the right-hand side. This process normally takes 10-15 minutes.

7. Once everything is up and running your screen should look similar to this.

You now have deployed Dagster onto a Kubernetes cluster using Plural.

Accessing your Dagster Deployment

Once in your Plural console, click on the launch button next to Dagster to access your Dagster installation. If you set up Plural OIDC earlier you don’t need to worry about managing authentication for logging into the console or the Dagster UI.

Once hit Launch and you allow access, you'll enter your Dagster UI to schedule your DAGs.

Accessing your Plural Console

If you set up Plural OIDC earlier you don’t need to worry about managing authentication for logging into the console or the Dagster UI.

The Plural Console is the operational hub for all applications managed by Plural. It is deployed in a cluster alongside applications and provides a few key features:

• Runbooks: Recommended settings and optimal operating procedures for running your application.
• Components: Statuses for each component in your Airbyte deployment and Kubernetes cluster. Inside this, you can look into Pod logs and events and use them to drill down into the root cause of any problems with your Kubernetes cluster.
• Nodes: Graphs and detailed information about the utilization of resources and deployments on each node.
• Incidents: View incidents that Plural automatically creates on component failure. Access a direct connection with our support team to troubleshoot any issues.
• Dashboards: View charts that have been tailored for your application. Every Plural application will ship with its own custom console dashboard.
  

Another important thing to call out is how to give a user access to your Dagster installation. Once you are back in the console you should click on the Plural account button in the upper right corner.

Next, click on users on the left side of the screen and then select invite user.

You’ll then be able to enter the user's email and we’ll send them a link to join your account and access the Dagster installation.

Next Steps with Plural

Through this article, you have learned how to:

• Create a Plural Git repository to store your infrastructure configuration
• Provision a fully configured Kubernetes cluster with no management experience necessary
• Install an instance of Dagster on your fresh Kubernetes cluster
• Invite a user to your Dagster installation

Are you looking to get your Dagster instance up and running on Kubernetes with minimal effort?

Reach out to me and the rest of the team over at Plural to learn more about how Plural works and how we are helping engineering teams across the world deploy open-source applications in a cloud production environment!

Make sure to join our Discord community for deployment help, discussion, and meeting other Plural users.

Ready to effortlessly deploy and operate open-source applications in minutes? Get started with Plural today.