From 9703c79813343deceb8dc898e4ee10c24b1f26c9 Mon Sep 17 00:00:00 2001 From: Tim Castillo Date: Wed, 3 Apr 2024 13:29:33 -0400 Subject: [PATCH] [du-dbt] adding pre-reqs and detail to the deployment step --- .../pages/dagster-dbt/lesson-2/1-requirements.md | 12 ++++++++++++ .../4-creating-the-manifest-during-deployment.md | 10 ++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-2/1-requirements.md b/docs/dagster-university/pages/dagster-dbt/lesson-2/1-requirements.md index 4b4e0a3f8c0b0..0c05c94ce2ad7 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-2/1-requirements.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-2/1-requirements.md @@ -21,6 +21,18 @@ To complete this course, you’ll need: pip --version ``` +## Pre-requisite experience + +- **dbt familiarity** An understanding of the basics of dbt is required to complete this course. We recommend that you first complete dbt's Fundamentals course if you're a new dbt user. + +- **Dagster familiarity** You'll need to know the basics of Dagster to complete this course. If you've never used Dagster before or want a refresher before getting started, check out the Dagster Essentials course. + +- **Python & SQL knowledge** While you don’t need to be a Python expert to get started, you do need some Python familiarity to complete this course and use Dagster. Here are some Pythonic skills that you’ll be using, along with resources to learn about them: [Functions](https://realpython.com/defining-your-own-python-function/), [Packages and Modules](https://dagster.io/blog/python-packages-primer-1), [Decorators](https://realpython.com/primer-on-python-decorators/), and [Type Hints](https://dagster.io/blog/python-type-hinting). You won’t be writing complex SQL, but you will need to understand the concept of SELECT statements, what tables are, and how to make them. If you’d like a 5-minute crash course, here’s a short article and cheatsheet on using SQL. + +## Need help? + +If you'd like some assistance while working through this course, you can reach out to the Dagster community on [Slack](https://dagster.io/slack) in the `#dagster-university` channel. + --- ## Clone the Dagster University project diff --git a/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md b/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md index c2013fc49e613..c990ae5d11841 100644 --- a/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md +++ b/docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md @@ -6,9 +6,11 @@ lesson: '7' # Creating the manifest during deployment -To recap, our deployment failed in the last section because Dagster couldn’t find a dbt manifest file, which it needs to turn dbt models into Dagster assets. This is because we built this file by running `dbt parse` during local development. You ran this manually in Lesson 3 and improved the experience in Lesson 4. However, you'll also need to build your dbt manifest file during deployment. We recommend adopting CI/CD to automate this process. +To recap, our deployment failed in the last section because Dagster couldn’t find a dbt manifest file, which it needs to turn dbt models into Dagster assets. This is because we built this file by running `dbt parse` during local development. You ran this manually in Lesson 3 and improved the experience in Lesson 4. However, you'll also need to build your dbt manifest file during deployment, which will require a couple additional steps. We recommend adopting CI/CD to automate this process. -In this case, Dagster Cloud’s out-of-the-box `deploy.yml` GitHub Action isn’t aware that you’re also trying to deploy a dbt project with Dagster. +Building your manifest for your production deployment will will be needed for both open source and Dagster Cloud deployments. In this case, Dagster Cloud’s out-of-the-box `deploy.yml` GitHub Action isn’t aware that you’re also trying to deploy a dbt project with Dagster. + +Since your CI/CD will be running in a fresh environment, you'll need to install dbt and run `dbt deps` before building your manifest with `dbt parse`. To get our deployment working, we need to add a step to our GitHub Actions workflow that runs the dbt commands required to generate the `manifest.json`. Specifically, we need to run `dbt deps` and `dbt parse` in the dbt project, just like you did during local development. @@ -46,3 +48,7 @@ At this point, your dbt project will be successfully deployed onto Dagster Cloud - ![dbt models in the Asset graph in Dagster Cloud](/images/dagster-dbt/lesson-7/asset-graph.png) {% /table %} + +{% callout %} +💡 **Experiencing issues?** There are two ways to deploy Dagster Cloud Serverless and we only made changes to deploy dbt with the default option, called Fast Deploys with PEX (Python Executable). If you receive an erro message to turn off Fast Deploys, GitHub Actions will skip the steps to build your dbt project. We recommend staying with Fast Deploys, if possible. If your project works locally, then the most likely issue is that the path where Dagster is looking for your manifest in production is different than that of the course. This can be solved by tinkering with where you're writing the manifest. If you'd like more support, you can reach out to the Dagster community on [Slack](https://dagster.io/slack) in the `#dagster-university` channel. +{% /callout %}