Skip to content

Commit

Permalink
[daggy-u] [dbt] - Rename to Dagster+ (#21455)
Browse files Browse the repository at this point in the history
## Summary & Motivation

This PR renames `Dagster Cloud` to `Dagster+` in the Dagster University
content.

## How I Tested These Changes

👀
  • Loading branch information
erinkcochran87 authored Apr 29, 2024
1 parent 557627c commit a4c7102
Show file tree
Hide file tree
Showing 7 changed files with 29 additions and 30 deletions.
2 changes: 1 addition & 1 deletion docs/dagster-university/pages/dagster-dbt.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ title: Dagster + dbt
- Lesson 7: Deploying to Production
- [Overview](/dagster-dbt/lesson-7/1-overview)
- [Pushing the project to GitHub](/dagster-dbt/lesson-7/2-pushing-the-project-to-github)
- [Setting up Dagster Cloud](/dagster-dbt/lesson-7/3-setting-up-dagster-cloud)
- [Setting up Dagster+](/dagster-dbt/lesson-7/3-setting-up-dagster-cloud)
- [Creating the manifest during deployment](/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment)
- [Preparing for a successful run](/dagster-dbt/lesson-7/5-preparing-for-a-successful-run)
- [Capstone](/dagster-dbt/capstone)
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,6 @@ By the end of the course, you will:
- Create dbt models and load them into Dagster as assets
- Run dbt and store the transformed data in a DuckDB database
- Apply partitions to incremental dbt models
- Deploy the dbt + Dagster project to Dagster Cloud
- Deploy the Dagster project to Dagster+

If you get stuck or want to jump ahead, check out the [finished project here on GitHub](https://github.com/dagster-io/project-dagster-university/tree/module/dagster-and-dbt).
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ lesson: '7'

At this point, you have a fully integrated Dagster and dbt project! You’ve learned how to load dbt models as Dagster assets, create dependencies, add partitions, and execute and monitor the resulting pipeline in the Dagster UI.

In this lesson, we’ll deploy your Dagster+dbt project to have it running in both local and production environments. We’ll walk through some considerations involved in bundling your dbt project up with Dagster.
In this lesson, we’ll deploy your Dagster and dbt project to have it running in both local and production environments. We’ll walk through some considerations involved in bundling your dbt project up with Dagster.

You’ll learn how to deploy your unified Dagster and dbt project to production, including pushing the project to GitHub and setting up CI/CD to factor in your dbt project. We’ll use Dagster Cloud because it’s a standardized and controlled experience that we can walk you through, but all of the general patterns can be applied to however you deploy Dagster.
You’ll learn how to deploy your unified Dagster and dbt project to production, including pushing the project to GitHub and setting up CI/CD to factor in your dbt project. We’ll use Dagster+ because it’s a standardized and controlled experience that we can walk you through, but all of the general patterns can be applied to however you deploy Dagster.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ lesson: '7'

# Pushing the project to GitHub

We’ll be using GitHub in this lesson because Dagster Cloud has a native integration with GitHub to quickly get deployment set up. This functionality can be easily replicated if your company uses a different version control provider, but we’ll standardize on using GitHub for now. Whether you use the command line or an app like GitHub Desktop is up to you.
We’ll be using GitHub in this lesson because Dagster+ has a native integration with GitHub to quickly get deployment set up. This functionality can be easily replicated if your company uses a different version control provider, but we’ll standardize on using GitHub for now. Whether you use the command line or an app like GitHub Desktop is up to you.

1. Because you cloned this project, it’ll already have a git history and context. Let’s delete that by running `rm -rf .git`.
2. Create a new repository on GitHub.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,36 +1,36 @@
---
title: "Lesson 7: Setting up Dagster Cloud"
title: "Lesson 7: Setting up Dagster+"
module: 'dbt_dagster'
lesson: '7'
---

# Setting up Dagster Cloud
# Setting up Dagster+

Now that the project is set up and ready in GitHub, it’s time to move to Dagster Cloud. To keep things simple, we’ll use a [Serverless deployment](https://docs.dagster.io/dagster-cloud/deployment/serverless) to deploy our project. This option offloads managing the required infrastructure to Dagster Labs.
Now that the project is set up and ready in GitHub, it’s time to move to Dagster+. To keep things simple, we’ll use a [Serverless deployment](https://docs.dagster.io/dagster-plus/deployment/serverless) to deploy our project. This option offloads managing the required infrastructure to Dagster Labs.

1. Sign up for a new [Dagster Cloud trial account](https://dagster.cloud/signup). Even if you already have an account, create a new one for this course. **Note:** When you sign up for a new account, you’ll automatically begin a free trial. You won’t be charged for anything after the trial unless you enter a credit card.
1. Sign up for a new [Dagster+ trial account](https://dagster.cloud/signup). Even if you already have an account, create a new one for this course. **Note:** When you sign up for a new account, you’ll automatically begin a free trial. You won’t be charged for anything after the trial unless you enter a credit card.
2. Complete the signup flow by creating an organization and finishing your user profile.
3. When prompted to select a deployment type, click **Serverless.**
4. The next step is to add our project to Dagster Cloud! Click the **Import a Dagster project** option and do the following:
4. The next step is to add our project to Dagster+! Click the **Import a Dagster project** option and do the following:
1. In the **Git scope** field, select the GitHub account or organization that contains your project repository.

{% callout %}
> 💡 **Don’t see the right account/organization?** You may need to install the Dagster Cloud GitHub app first. To do this, click **+ Add account or organization.** You’ll be redirected to GitHub to complete the setup, and then automatically sent back to Dagster Cloud when finished. If you’re installing within your company’s GitHub organization, you may need your company’s GitHub admin to approve the app.
> 💡 **Don’t see the right account/organization?** You may need to install the Dagster+ GitHub app first. To do this, click **+ Add account or organization.** You’ll be redirected to GitHub to complete the setup, and then automatically sent back to Dagster+ when finished. If you’re installing within your company’s GitHub organization, you may need your company’s GitHub admin to approve the app.
{% /callout %}

2. In the **Repository** field, select the repository containing your Dagster project.
3. Click **Deploy. Note that the deployment can take a few minutes.** Feel free to go grab a snack while you’re waiting!

---

## What happens when Dagster Cloud deploys code?
## What happens when Dagster+ deploys code?

When Dagster deploys the code, a few things happen:

- Dagster creates a new code location for the repository in Dagster Cloud in the `prod` deployment
- Dagster creates a new code location for the repository in Dagster+ in the `prod` deployment
- Dagster adds two GitHub Action files to the repository:
- `.github/workflows/deploy.yml` - This file sets up Continuous Deployment (CD) for the repository. We won’t talk through all the steps here, but a high-level summary is that every time a change is made to the `main` branch of your repository, this GitHub Action will build your Dagster project and deploy it to Dagster Cloud.
- `.github/workflows/branch_deployments.yml` - This file enables the use of [Branch Deployments](https://docs.dagster.io/dagster-cloud/managing-deployments/branch-deployments), a Dagster Cloud feature that automatically creates staging environments for your Dagster code with every pull request. We won’t work with Branch Deployments during this lesson, but we highly recommend trying them out!
- `.github/workflows/deploy.yml` - This file sets up Continuous Deployment (CD) for the repository. We won’t talk through all the steps here, but a high-level summary is that every time a change is made to the `main` branch of your repository, this GitHub Action will build your Dagster project and deploy it to Dagster+.
- `.github/workflows/branch_deployments.yml` - This file enables the use of [Branch Deployments](https://docs.dagster.io/dagster-cloud/managing-deployments/branch-deployments), a Dagster+ feature that automatically creates staging environments for your Dagster code with every pull request. We won’t work with Branch Deployments during this lesson, but we highly recommend trying them out!

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ lesson: '7'

To recap, our deployment failed in the last section because Dagster couldn’t find a dbt manifest file, which it needs to turn dbt models into Dagster assets. This is because we built this file by running `dbt parse` during local development. You ran this manually in Lesson 3 and improved the experience in Lesson 4. However, you'll also need to build your dbt manifest file during deployment, which will require a couple additional steps. We recommend adopting CI/CD to automate this process.

Building your manifest for your production deployment will will be needed for both open source and Dagster Cloud deployments. In this case, Dagster Cloud’s out-of-the-box `deploy.yml` GitHub Action isn’t aware that you’re also trying to deploy a dbt project with Dagster.
Building your manifest for your production deployment will will be needed for both open source and Dagster+ deployments. In this case, Dagster+’s out-of-the-box `deploy.yml` GitHub Action isn’t aware that you’re also trying to deploy a dbt project with Dagster.

Since your CI/CD will be running in a fresh environment, you'll need to install dbt and run `dbt deps` before building your manifest with `dbt parse`.

Expand All @@ -35,7 +35,7 @@ To get our deployment working, we need to add a step to our GitHub Actions workf
Once the new step is pushed to the remote, GitHub will automatically try to run a new job using the updated workflow.
At this point, your dbt project will be successfully deployed onto Dagster Cloud and you should be able to see your models in the asset graph!
At this point, your dbt project will be successfully deployed onto Dagster+ and you should be able to see your models in the asset graph!
{% table %}
Expand All @@ -44,16 +44,16 @@ At this point, your dbt project will be successfully deployed onto Dagster Cloud
---
- ![Successful deployment screen in Dagster Cloud](/images/dagster-dbt/lesson-7/successful-cloud-setup.png)
- ![dbt models in the Asset graph in Dagster Cloud](/images/dagster-dbt/lesson-7/asset-graph.png)
- ![Successful deployment screen in Dagster+](/images/dagster-dbt/lesson-7/successful-cloud-setup.png)
- ![dbt models in the Asset graph in Dagster+](/images/dagster-dbt/lesson-7/asset-graph.png)
{% /table %}
---
## Experiencing issues?
There are two ways to deploy Dagster Cloud Serverless. In our case, we only made changes to deploy dbt with the default option, called Fast Deploys with PEX (Python Executable).
There are two ways to deploy Dagster+ Serverless. In our case, we only made changes to deploy dbt with the default option, called Fast Deploys with PEX (Python Executable).
**If you receive an error message to turn off Fast Deploys**, GitHub Actions will skip the steps that build the dbt project. We recommend staying with Fast Deploys, if possible.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ lesson: '7'
# Preparing for a successful run

{% callout %}
>💡 **Heads up! This section is optional.** The goal of this lesson was to teach you how to successfully **deploy** to Dagster Cloud, which you completed in the last section. Preparing for a successful run in Dagster Cloud requires using some external services, which may not translate to the external services you prefer to use. As such, we’ve opted to make this section optional.
>💡 **Heads up! This section is optional.** The goal of this lesson was to teach you how to successfully **deploy** to Dagster+, which you completed in the last section. Preparing for a successful run in Dagster+ requires using some external services, which may not translate to the external services you prefer to use. As such, we’ve opted to make this section optional.
{% /callout %}

In previous lessons, you followed along by adding our example code to your local project. You successfully materialized the assets in the project and stored the resulting data in a local DuckDB database.
Expand All @@ -25,13 +25,13 @@ Since you'll be deploying your project in production, you'll need production sys

The code you cloned in the starter project already has some logic to dynamically switch between local and cloud storage, along with the paths to reference. To trigger the switch, you can set an environment variable called `DAGSTER_ENVIRONMENT` and set it to `prod`. This will tell the pipeline to use the production paths and storage.

In summary, before you can run this pipeline in Dagster Cloud, you’ll need to:
In summary, before you can run this pipeline in Dagster+, you’ll need to:

1. Set up an S3 bucket to store the files/assets that we download and generate
2. Sign up for a free Motherduck account to replace our local DuckDB instance
3. Connect an S3 user with access to the S3 bucket to the Motherduck account
4. Add a new production target to the dbt project
5. Add the environment variables for the S3 user and Motherduck token to Dagster Cloud
5. Add the environment variables for the S3 user and Motherduck token to Dagster+

We’ll show you how to do 4 and 5 so you can do this with your credentials when you’re ready.

Expand Down Expand Up @@ -63,7 +63,7 @@ Because we’re still using a DuckDB-backed database, our `type` will also be `d

## Adding a prod target to deploy.yml

Next, we need to update the dbt commands in the `.github/workflows/deploy.yml` file to target the new `prod` profile. This will ensure that dbt uses the correct connection details when the GitHub Action runs as part of our Dagster Cloud deployment.
Next, we need to update the dbt commands in the `.github/workflows/deploy.yml` file to target the new `prod` profile. This will ensure that dbt uses the correct connection details when the GitHub Action runs as part of our Dagster+ deployment.

Open the file, scroll to the dbt step you added, and add `-- target prod` after the `dbt parse` command. This command should be on or around line 52:

Expand All @@ -83,13 +83,13 @@ Save and commit the file to git. Don’t forget to push to remote!

---

## Adding environment variables to Dagster Cloud
## Adding environment variables to Dagster+

The last step in preparing for a successful run is to move environment variables to Dagster Cloud. These variables were available to us via the `.env` file while we were working locally, but now that we’ve moved to a different environment, we’ll need to make them accessible again.
The last step in preparing for a successful run is to move environment variables to Dagster+. These variables were available to us via the `.env` file while we were working locally, but now that we’ve moved to a different environment, we’ll need to make them accessible again.

### Environment variables

The following table contains the environment variables we need to create in Dagster Cloud:
The following table contains the environment variables we need to create in Dagster+:

{% table %}

Expand Down Expand Up @@ -130,7 +130,7 @@ The following table contains the environment variables we need to create in Dags

### Creating environment variables

1. In the Dagster Cloud UI, click **Deployment > Environment variables**.
1. In the Dagster+ UI, click **Deployment > Environment variables**.
2. Click the **Add environment variable** button on the right side of the screen.
3. In the **Create environment variable** window, fill in the following:
1. **Name** - The name of the environment variable. For example: `DUCKDB_DATABASE`
Expand All @@ -144,5 +144,4 @@ Repeat these steps until all the environment variables have been added.

## Running the pipeline

At this point, you're ready to run the pipeline in production! Navigate to the asset graph, click **Materialize all**, and watch as it all comes together:

At this point, you're ready to run the pipeline in production! Navigate to the asset graph, click **Materialize all**, and watch as it all comes together.

2 comments on commit a4c7102

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-2vnabej1u-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit a4c7102.
This pull request is being automatically deployed with vercel-action

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-university ready!

✅ Preview
https://dagster-university-3c7jkojgh-elementl.vercel.app

Built with commit a4c7102.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.