diff --git a/website/blog/2024-07-24-cityblock-merge-jobs.md b/website/blog/2024-07-24-cityblock-merge-jobs.md deleted file mode 100644 index 228ab17615f..00000000000 --- a/website/blog/2024-07-24-cityblock-merge-jobs.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: "Practitioner Q&A: How Cityblock Health adopted dbt Cloud's Merge Jobs to deploy new code faster" -description: "Cityblock Health swapped their GitHub Actions for native dbt Cloud functionality to streamline their project deployments and avoid redundant resouce consumption." -slug: cityblock-merge-jobs -authors: [nathaniel_burren, joel_labes] -tags: [dbt Cloud] -hide_table_of_contents: false -date: 2024-07-30 -is_featured: true ---- - -dbt Labs recently added support for a new type of job in dbt Cloud, called a **[Merge Job](/docs/deploy/merge-jobs)**. Just like a Continuous Integration job is triggered based on a new Pull Request being opened in your git repository, the Merge Job triggers once code is merged into your environment’s configured branch (e.g. `main` for the Production environment, or `uat` for the Staging environment). - -Triggering a run when code is merged enables your team to practice Continuous Deployment, getting changed models deployed as quickly as possible and reducing resource consumption from unnecessary overbuilding. Alternatively, just kick off a `dbt compile` to update your project’s state for Slim CI comparisons. Check out the [Trigger on Merge docs](/docs/deploy/merge-jobs) for full details on enabling this in your project. - -The Cityblock Health team were one of the first companies to use Merge Jobs during the beta period, and I (Joel) caught up with Analytics Engineer Nathaniel Burren to hear how they did it and what benefits they’ve seen. - - - -## How long have you been using dbt at Cityblock? - -Cityblock chose dbt in 2019 as the tool to make all of our SQL based analysis version controlled, composable, and reusable. However, over the years our monolithic project turned into a jungle of over 2,200 models. This made us rethink our approach and move to dbt Cloud while also adapting a [multi-project methodology](/best-practices/how-we-mesh/mesh-1-intro). We presented [our migration story at Coalesce last year](https://www.youtube.com/watch?v=oO7whNtd9Jg) and go into more details there. - -Our hope is to create core assets maintained by our Analytics Engineers that our downstream teams can use in their own projects. This also allows us to retain control of these core assets as our single sources of truth while the downstream projects can create their own assets based on their subject matter expertise (that in turn other projects can use, instead of having a mess of duplicate models). - -## What made you excited enough about Merge Jobs to sign up for the beta? - -Our goal when adopting Merge Jobs was to get model changes in our Staging and Production environments deployed as fast as possible. We also like to have fresh Explorer data as we have many eyes from downstream users looking at the core models in our data platform. Keeping the environments up to date with our git repo means that when we fix something the resolution is immediately deployed, not languishing waiting for our overnight build. - -## What were you doing to achieve these goals before? - -We used to use a GitHub Action to trigger a production run on a merge to main. However most of our Analytics Engineerss don’t understand GHAs and we would have to get our DevOps friends to help us whenever configuration changes were needed. In turn that would cause us to have to wait on them or an AE that knows GHAs to help change/maintain the code. - -## And what are you doing now? How did you configure your Merge Job? - -Because we have so many dbt projects split across different domains, one of my colleagues built a Terraform module which builds new dbt Cloud projects, including environments, jobs, and permissions. We added the Merge Job to that definition and deployed it everywhere in one go. - -The job itself only has one step, `dbt build --select state:modified+`. It defers to itself, so every time a commit is merged to main the only things that get built are models that changed compared to that last run. - -## How has switching to the native functionality improved things for you? - -The biggest place this has helped us is no longer having to babysit deploy jobs. With the fast pace we’re working at right now, it really helps us keep working on other things and not worry about manually triggering jobs. - -## What advice would you give to other teams who are implementing Merge Jobs? - -Make sure you’ve got your [branch protection rules in GitHub](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule#creating-a-branch-protection-rule) locked down. We noticed that some downstream project users were merging PRs even though their [dbt-project-evaluator checks](/blog/align-with-dbt-project-evaluator) were failing. Since failed jobs aren’t eligible for deferral, this caused some weird behavior for us until we sorted it out. diff --git a/website/dbt-versions.js b/website/dbt-versions.js index 7709fea304c..5ad3a3048c5 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -15,10 +15,6 @@ exports.versions = [ version: "1.6", EOLDate: "2024-07-31", }, - { - version: "1.5", - EOLDate: "2024-04-27", - }, ] exports.versionedPages = [ diff --git a/website/docs/community/spotlight/meagan-palmer.md b/website/docs/community/spotlight/meagan-palmer.md new file mode 100644 index 00000000000..ff45a3d6b7d --- /dev/null +++ b/website/docs/community/spotlight/meagan-palmer.md @@ -0,0 +1,41 @@ +--- +id: meagan-palmer +title: Meagan Palmer +description: | + I first started using dbt in 2016 or 2017 (I can't remember exactly). Since then, I have moved into data and analytics consulting and have dipped in and out of the dbt Community. + Late last year, I started leading dbt Cloud training courses and spending more time in the dbt Slack. + In consulting, I get to use a range of stacks. I've used dbt with Redshift, Snowflake, and Databricks in production settings with a range of loaders & reporting tools, and I've been enjoying using DuckDB for some home experimentation. + To share some of the experiences, I regularly post to LinkedIn and have recently started Analytics Engineering Today, a twice monthly newsletter about dbt in practice. +image: /img/community/spotlight/Meagan-Palmer.png +pronouns: she/her +location: Sydney, Australia +jobTitle: Principal Consultant +companyName: Altis Consulting +socialLinks: + - name: LinkedIn + link: https://www.linkedin.com/in/meaganpalmer/ +dateCreated: 2024-07-29 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I was fortunate that Jon Bradley at Nearmap had the vision to engage the then Fishtown Analytics team (as the dbt Labs team was formerly called) as consultants and begin using dbt in our stack. I can't thank him enough. It was a turning point for my career, where I could combine my interests and experiences in delivering business value, data, product management, and software engineering practices. + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +Being in Australia, I often see replies from Jeremy Yeo to people in the dbt Slack. His clarity of communication is impressive. + +For growth, I'm hoping that others can benefit from the wide range of experience I have. My newsletter, Analytics Engineering Today on LinkedIn aims to upskill the dbt Community and shed some light on some useful features that might not be well known. + +I'll be at Coalesce and am doing some webinars/events later in the year. Come say hi, I love talking dbt and analytics engineering with people. + +## What have you learned from community members? What do you hope others can learn from you? + +The community members are amazing. It's great to be among a group of people that want to learn and improve. +I've learned a lot - both from other members helping with my queries and in reading how other businesses have implemented dbt, including their stories on the organizational & technical issues they face. + +I hope I can help instill a sense that simple, clean solutions are possible and preferable. I want to highlight that it is important to focus on what is the actual problem you are trying to solve and the fact that it's worth asking for help when you're starting to get stuck. + +I'm keen to help more women get the courage to work & lead in STEM. There has been a lot of progress made over the course of my career which is great to see. Australian/NZ women, please connect with me, happy to chat. diff --git a/website/docs/community/spotlight/mikko-sulonen.md b/website/docs/community/spotlight/mikko-sulonen.md new file mode 100644 index 00000000000..689b53db97e --- /dev/null +++ b/website/docs/community/spotlight/mikko-sulonen.md @@ -0,0 +1,33 @@ +--- +id: mikko-sulonen +title: Mikko Sulonen +description: | + I've been working with data since 2016. I first started with the on-prem SQL Server S-stack of SSIS, SSAS, SSRS. I did some QlikView and Qlik Sense, and some Power BI. Nowadays, I work mostly with Snowflake, Databricks, Azure, and dbt, of course. While tools and languages have come and gone, SQL has stayed. I've been a consultant for all of my professional life. +image: /img/community/spotlight/Mikko-Sulonen.png +pronouns: he/him +location: Tampere, Finland +jobTitle: Data Architect, Partner +companyName: Recordly Oy +socialLinks: + - name: LinkedIn + link: https://www.linkedin.com/in/mikkosulonen/ +dateCreated: 2024-07-28 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I started using dbt around 2019-2020, I think. Snapshots (no longer "archives") were a new thing along with the RPC server! I asked around my then-company: pretty much nobody had used dbt, though some commented that it did look promising. That left me looking elsewhere for experiences and best practices around the tool, and I found different blog writers and eventually the [dbt Slack](https://www.getdbt.com/community/join-the-community). I quickly noticed I could learn much more from the experiences of others than by trying everything myself. After just lurking for a while, I started to answer people's questions and give my own thoughts. This was completely new to me: voicing my input and opinions to people I had never met or who were not my colleagues. + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +There are quite many. I started to write some names here, but felt the list was getting a bit long and I'd still forget somebody! What the community leaders have in common is that they are approachable, knowledgeable, and passionate. They want to help others, they want to drive the community forward, and they are down to earth. I've had the pleasure of meeting many of them in person at the past two [Coalesces](https://coalesce.getdbt.com/), and I hope to meet many more! + +Growing my own leadership in the community... That's an interesting question: I hadn't really identified myself as leader in the community before. Maybe I should come out of the Slack channels and join and/or organize some [dbt Meetups](https://www.meetup.com/pro/dbt/)? I always try to answer even the simplest questions, even if they've been answered a hundred times already. Every day, new people are introduced to dbt, and they are facing issues for the first time. Every one of us was new at one point! + +## What have you learned from community members? What do you hope others can learn from you? + +I've learnt a surprising amount about the different teams and ways of working with and around data. I've learnt that it is highly probable that somebody somewhere has already had, and likely solved, the problem you are having. All that is needed to connect the dots is for people to speak and listen. + +When asking and answering questions, I try to hone in on what they're really trying to get at. I ask, why is it that you want to do something in that particular way? I like to say, don't love the solutions, love the problems! diff --git a/website/docs/community/spotlight/radovan-bacovic.md b/website/docs/community/spotlight/radovan-bacovic.md new file mode 100644 index 00000000000..186edccac6d --- /dev/null +++ b/website/docs/community/spotlight/radovan-bacovic.md @@ -0,0 +1,42 @@ +--- +id: radovan-bacovic +title: Radovan Bacovic +description: | + My professional journey and friendship with data started 20 years ago. + I've experienced many tools and modalities: from good old RDMS systems and various programming languages like Java and C# in the early days of my career, through private clouds, to MPP databases and multitier architecture. I also saw the emergence of cloud technology, and experienced the changes that came with it, up to the contemporary approach to data, which includes AI tools and practices. I am still excited to get new knowledge and solve problems in the data world. I always enjoy using SQL and Python as my primary "weapons" of choice together with other building blocks for successful data products like Snowflake, dbt, Tableau, Fivetran, Stich, Monte Carlo, DuckDB and more. + I consider myself as an experienced data engineer and a wannabe "best bad Conference speaker." +image: /img/community/spotlight/Radovan-Bacovic.png +pronouns: he/him +location: Novi Sad, Serbia +jobTitle: Staff Data Engineer +companyName: GitLab +organization: Permifrost maintainer +socialLinks: + - name: X + link: https://x.com/svenmurtinson + - name: LinkedIn + link: https://www.linkedin.com/in/radovan-ba%C4%87ovi%C4%87-6498603/ + - name: Gitlab Profile + link: https://gitlab.com/rbacovic +dateCreated: 2024-07-28 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I have been in the dbt Community for almost 3 years now. The biggest impact that dbt has had on my professional journey is that it has given me a trustworthy partner for data transformation, and a single source of truth for all our data modification needs. As a public speaker who travels internationally, I recognized the interest of the data community in dbt around the world and, in response, organised several workshops and talks to help people use dbt. Let's just say that jumping into a great partnership with the dbt Community has been the greatest takeaway for me! + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +A leader from the dbt Commuity who I have found to be the most prominent is Sean McIntyre from dbt Labs, as I've had the privilege to collaborate with him many times. We recognized that we had a similar passion and energy; it looks like we are on the same journey. + +I wanted to be more involved in the dbt Community, spread the word, and share my journey through tutorials, conference talks, blogs and meetups. I think I am capable of addressing my influence in that direction. I am also interested in extending the dbt functionality and automating the deployment, testing and execution of dbt. In other words, I try to find good ways to leverage using DevSecOps for dbt to make our development faster and make dbt our trustworthy partner. + +## What have you learned from community members? What do you hope others can learn from you? + +Being part of the community is always a two-way street and always has a positive result for all of us. Great ideas and sharing vision and energy are the number one things. On the other side, I always find the quick and "best in class" answer to my questions from the community and try to return the favour and be helpful whenever possible. As I said, through talks and tutorials, I think I am the most beneficial community member in that role. + +## Anything else interesting you want to tell us? + +Nothing to add. Still passionate about discovering the tool and can't wait to see what the AI uprising will bring to us! diff --git a/website/docs/docs/build/custom-aliases.md b/website/docs/docs/build/custom-aliases.md index 9e9f91f968d..ee56480dc5c 100644 --- a/website/docs/docs/build/custom-aliases.md +++ b/website/docs/docs/build/custom-aliases.md @@ -73,8 +73,6 @@ To override dbt's alias name generation, create a macro named `generate_alias_na The default implementation of `generate_alias_name` simply uses the supplied `alias` config (if present) as the model alias, otherwise falling back to the model name. This implementation looks like this: - - ```jinja2 @@ -100,8 +98,6 @@ The default implementation of `generate_alias_name` simply uses the supplied `al - - import WhitespaceControl from '/snippets/_whitespace-control.md'; diff --git a/website/docs/docs/build/custom-databases.md b/website/docs/docs/build/custom-databases.md index 2d430033b66..7a607534230 100644 --- a/website/docs/docs/build/custom-databases.md +++ b/website/docs/docs/build/custom-databases.md @@ -89,14 +89,10 @@ import WhitespaceControl from '/snippets/_whitespace-control.md'; - - ### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) - - ## Considerations ### BigQuery diff --git a/website/docs/docs/build/custom-schemas.md b/website/docs/docs/build/custom-schemas.md index 846f2d1c341..6dabf56943c 100644 --- a/website/docs/docs/build/custom-schemas.md +++ b/website/docs/docs/build/custom-schemas.md @@ -154,14 +154,10 @@ The following context methods _are_ available in the `generate_schema_name` macr Globally-scoped variables and variables defined on the command line with [--vars](/docs/build/project-variables) are accessible in the `generate_schema_name` context. - - ### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) - - ## A built-in alternative pattern for generating schema names A common customization is to ignore the target schema in production environments, and ignore the custom schema configurations in other environments (such as development and CI). diff --git a/website/docs/docs/build/data-tests.md b/website/docs/docs/build/data-tests.md index 38a2c74b646..158b8427136 100644 --- a/website/docs/docs/build/data-tests.md +++ b/website/docs/docs/build/data-tests.md @@ -19,7 +19,7 @@ keywords: :::important -In dbt v1.8, what was previously known as "tests" are now called "data tests" with the addition of [unit tests](/docs/build/unit-tests). The YAML key `tests:` is still supported as an alias for data tests but will be deprecated in the future in favor of `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. +From dbt v1.8, "tests" are now called "data tests" to disambiguate from [unit tests](/docs/build/unit-tests). The YAML key `tests:` is still supported as an alias for `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. ::: @@ -275,13 +275,11 @@ In dbt version 1.8, we updated the `tests` configuration to `data_tests`. For de - + -Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, it was necessary to update our naming conventions and syntax. - -As of v1.8, `tests:` is still supported in your YML configuration files as an alias but will be deprecated in the future in favor of `data_tests:`. +Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, the key was renamed from `tests:` to `data_tests:`. -As we progress towards this deprecation, we'll update the examples in our docs pages to reflect this new syntax, but we highly recommend you begin the migration process as soon as you upgrade to v1.8 to avoid interruptions or issues in the future. +dbt still supports `tests:` in your YML configuration files for backwards-compatibility purposes, and you might see it used throughout our documentation. However, you can't have a `tests` and a `data_tests` key associated with the same resource (e.g. a single model) at the same time. @@ -306,6 +304,8 @@ data_tests: +To suppress warnings about the rename, add `TestsConfigDeprecation` to the `silence` block of the `warn_error_options` flag in `dbt_project.yml`, [as described in the Warnings documentation](https://docs.getdbt.com/reference/global-configs/warnings). + ## FAQs diff --git a/website/docs/docs/build/environment-variables.md b/website/docs/docs/build/environment-variables.md index 664a7a6cf15..01601ce7eb8 100644 --- a/website/docs/docs/build/environment-variables.md +++ b/website/docs/docs/build/environment-variables.md @@ -9,7 +9,7 @@ Environment variables can be used to customize the behavior of a dbt project dep :::info Environment Variable Naming and Prefixing -Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET` or `DBT_ENV_CUSTOM_ENV_`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. +Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET` or `DBT_ENV_CUSTOM_ENV_`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. ::: @@ -86,7 +86,7 @@ There are some known issues with partial parsing of a project and changing envir ### Handling secrets -While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in the UI, you can prefix the key with `DBT_ENV_SECRET_``DBT_ENV_SECRET`. This functionality is supported from `dbt v1.0` and on. +While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in the UI, you can prefix the key with `DBT_ENV_SECRET`. This functionality is supported from `dbt v1.0` and on. @@ -97,8 +97,6 @@ While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud ha dbt Cloud has a number of pre-defined variables built in. Variables are set automatically and cannot be changed. - - **dbt Cloud IDE details** The following environment variable is set automatically for the dbt Cloud IDE: @@ -111,8 +109,6 @@ The following environment variable is set automatically for the dbt Cloud IDE: Use case — This is useful in cases where you want to dynamically use the Git branch name as a prefix for a [development schema](/docs/build/custom-schemas) ( `{{ env_var ('DBT_CLOUD_GIT_BRANCH') }}` ). - - **dbt Cloud context** The following environment variables are set automatically for deployment runs: diff --git a/website/docs/docs/build/incremental-strategy.md b/website/docs/docs/build/incremental-strategy.md index 09a9187f8aa..8e86da0eba8 100644 --- a/website/docs/docs/build/incremental-strategy.md +++ b/website/docs/docs/build/incremental-strategy.md @@ -19,22 +19,6 @@ Click the name of the adapter in the below table for more information about supp The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in dbt v1.6. - - -| data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | -|-----------------------------------------------------------------------------------------------------|:--------:|:-------:|:---------------:|:------------------:| -| [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | ✅ | | ✅ | | -| [dbt-redshift](/reference/resource-configs/redshift-configs#incremental-materialization-strategies) | ✅ | | ✅ | | -| [dbt-bigquery](/reference/resource-configs/bigquery-configs#merge-behavior-incremental-models) | | ✅ | | ✅ | -| [dbt-spark](/reference/resource-configs/spark-configs#incremental-models) | ✅ | ✅ | | ✅ | -| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) | ✅ | ✅ | | ✅ | -| [dbt-snowflake](/reference/resource-configs/snowflake-configs#merge-behavior-incremental-models) | ✅ | ✅ | ✅ | | -| [dbt-trino](/reference/resource-configs/trino-configs#incremental) | ✅ | ✅ | ✅ | | - - - - - | data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | |-----------------------------------------------------------------------------------------------------|:--------:|:-------:|:---------------:|:------------------:| | [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | ✅ | ✅ | ✅ | | @@ -46,7 +30,6 @@ The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in | [dbt-trino](/reference/resource-configs/trino-configs#incremental) | ✅ | ✅ | ✅ | | | [dbt-fabric](/reference/resource-configs/fabric-configs#incremental) | ✅ | | ✅ | | - :::note Snowflake Configurations diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md index 5e0772f517e..9458487e8d4 100644 --- a/website/docs/docs/build/measures.md +++ b/website/docs/docs/build/measures.md @@ -141,13 +141,13 @@ semantic_models: description: Distinct count of transactions expr: transaction_id agg: count_distinct - - name: transactions + - name: transaction_amount_avg description: The average value of transactions expr: transaction_amount_usd agg: average - name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition description: The total USD value of valid transactions only - expr: CASE WHEN is_valid = True then 1 else 0 end + expr: CASE WHEN is_valid = True then transaction_amount_usd else 0 end agg: sum - name: transactions description: The average value of transactions. diff --git a/website/docs/docs/build/packages.md b/website/docs/docs/build/packages.md index 7871e70239e..0b69d10cee6 100644 --- a/website/docs/docs/build/packages.md +++ b/website/docs/docs/build/packages.md @@ -32,7 +32,7 @@ import UseCaseInfo from '/snippets/_packages_or_dependencies.md'; ## How do I add a package to my project? -1. Add a file named `dependencies.yml` or `packages.yml` to your dbt project. This should be at the same level as your `dbt_project.yml` file. +1. Add a file named `dependencies.yml` or `packages.yml` to your dbt project. This should be at the same level as your `dbt_project.yml` file. 2. Specify the package(s) you wish to add using one of the supported syntaxes, for example: @@ -195,7 +195,7 @@ This method allows the user to clone via HTTPS by passing in a git token via an :::info dbt Cloud usage -If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. +If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. ::: In GitHub: diff --git a/website/docs/docs/build/project-variables.md b/website/docs/docs/build/project-variables.md index 8ed0d86aa9e..6e1cc717306 100644 --- a/website/docs/docs/build/project-variables.md +++ b/website/docs/docs/build/project-variables.md @@ -91,25 +91,12 @@ Variables defined with the `--vars` command line argument override variables def The order of precedence for variable declaration is as follows (highest priority first): - - 1. The variables defined on the command line with `--vars`. 2. The package-scoped variable declaration in the root `dbt_project.yml` file 3. The global variable declaration in the root `dbt_project.yml` file 4. If this node is defined in a package: variable declarations in that package's `dbt_project.yml` file 5. The variable's default argument (if one is provided) - - - - -1. The variables defined on the command line with `--vars` -2. The package-scoped variable declaration in the root `dbt_project.yml` file -3. The global variable declaration in the root `dbt_project.yml` file -4. The variable's default argument (if one is provided). - - - If dbt is unable to find a definition for a variable after checking all possible variable declaration places, then a compilation error will be raised. **Note:** Variable scope is based on the node ultimately using that variable. Imagine the case where a model defined in the root project is calling a macro defined in an installed package. That macro, in turn, uses the value of a variable. The variable will be resolved based on the _root project's_ scope, rather than the package's scope. diff --git a/website/docs/docs/build/snapshots.md b/website/docs/docs/build/snapshots.md index f7bdef407ba..82b5104fcef 100644 --- a/website/docs/docs/build/snapshots.md +++ b/website/docs/docs/build/snapshots.md @@ -593,6 +593,5 @@ Snapshot results: - diff --git a/website/docs/docs/cloud-integrations/about-snowflake-native-app.md b/website/docs/docs/cloud-integrations/about-snowflake-native-app.md index 8a703d69eba..fa1c54ca5a6 100644 --- a/website/docs/docs/cloud-integrations/about-snowflake-native-app.md +++ b/website/docs/docs/cloud-integrations/about-snowflake-native-app.md @@ -45,3 +45,6 @@ If you're interested, please [contact us](matilto:sales_snowflake_marketplace@db ## Support If you have any questions about the dbt Snowflake Native App, you may [contact our Support team](mailto:dbt-snowflake-marketplace@dbtlabs.com) for help. Please provide information about your installation of the Native App, including your dbt Cloud account ID and Snowflake account identifier. + +## Limitations +- The Native app does not support dbt Cloud accounts with [IP Restrictions](/docs/cloud/secure/ip-restrictions] enabled. diff --git a/website/docs/docs/cloud-integrations/avail-sl-integrations.md b/website/docs/docs/cloud-integrations/avail-sl-integrations.md index 30ca750a5c6..eea93c92b93 100644 --- a/website/docs/docs/cloud-integrations/avail-sl-integrations.md +++ b/website/docs/docs/cloud-integrations/avail-sl-integrations.md @@ -9,14 +9,6 @@ meta: api_name: dbt Semantic Layer APIs --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - There are a number of data applications that seamlessly integrate with the dbt Semantic Layer, powered by MetricFlow, from business intelligence tools to notebooks, spreadsheets, data catalogs, and more. These integrations allow you to query and unlock valuable insights from your data ecosystem. Use the [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) to simplify metric queries, optimize your development workflow, and reduce coding. This approach also ensures data governance and consistency for data consumers. diff --git a/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md b/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md index 23b6c8ca906..0ca07ba85cb 100644 --- a/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md +++ b/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md @@ -14,7 +14,7 @@ dbt Cloud is [hosted](/docs/cloud/about-cloud/architecture) in multiple regions |--------|----------|------------|--------------|----------------|-----------|-----------------| | North America [^1] | AWS us-east-1 (N. Virginia) | **Multi-tenant:** cloud.getdbt.com
**Cell based:** ACCOUNT_PREFIX.us1.dbt.com | 52.45.144.63
54.81.134.249
52.22.161.231
52.3.77.232
3.214.191.130
34.233.79.135 | ✅ | ✅ | ✅ | | EMEA [^1] | AWS eu-central-1 (Frankfurt) | emea.dbt.com | 3.123.45.39
3.126.140.248
3.72.153.148 | ❌ | ❌ | ✅ | -| EMEA [^1] | Azure
North Europe (Ireland) | **Cell based:** ACCOUNT_PREFIX.eu2.dbt.com (beta invite only) | 20.13.190.192/26 | ❌ | ❌ | ✅ | +| EMEA [^1] | Azure
North Europe (Ireland) | **Cell based:** ACCOUNT_PREFIX.eu2.dbt.com | 20.13.190.192/26 | ❌ | ❌ | ✅ | | APAC [^1] | AWS ap-southeast-2 (Sydney)| au.dbt.com | 52.65.89.235
3.106.40.33
13.239.155.206
| ❌ | ❌ | ✅ | | Virtual Private dbt or Single tenant | Customized | Customized | Ask [Support](/community/resources/getting-help#dbt-cloud-support) for your IPs | ❌ | ❌ | ✅ | diff --git a/website/docs/docs/cloud/about-cloud/tenancy.md b/website/docs/docs/cloud/about-cloud/tenancy.md index 5785533264e..efbc248f91e 100644 --- a/website/docs/docs/cloud/about-cloud/tenancy.md +++ b/website/docs/docs/cloud/about-cloud/tenancy.md @@ -10,7 +10,7 @@ import AboutCloud from '/snippets/_test-tenancy.md'; ### Multi-tenant -The Multi Tenant (SaaS) deployment environment refers to the SaaS dbt Cloud application hosted by dbt Labs. This is the most commonly used deployment and is completely managed and maintained by dbt Labs, the makers of dbt. As a SaaS product, a user can quickly [create an account](https://www.getdbt.com/signup/) on our North American servers and get started using the dbt and related services immediately. _If your organization requires cloud services hosted on EMEA or APAC regions_, please [contact us](https://www.getdbt.com/contact/). The deployment is hosted on AWS and is always kept up to date with the currently supported dbt versions, software updates, and bug fixes. +The Multi Tenant (SaaS) deployment environment refers to the SaaS dbt Cloud application hosted by dbt Labs. This is the most commonly used deployment and is completely managed and maintained by dbt Labs, the makers of dbt. As a SaaS product, a user can quickly [create an account](https://www.getdbt.com/signup/) on our North American servers and get started using the dbt and related services immediately. _If your organization requires cloud services hosted on EMEA or APAC regions_, please [contact us](https://www.getdbt.com/contact/). The deployments are hosted on AWS or Azure ([Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud)) and are always kept up to date with the currently supported dbt versions, software updates, and bug fixes. ### Single tenant diff --git a/website/docs/docs/cloud/configure-cloud-cli.md b/website/docs/docs/cloud/configure-cloud-cli.md index 88e7d39b369..2874e166a8f 100644 --- a/website/docs/docs/cloud/configure-cloud-cli.md +++ b/website/docs/docs/cloud/configure-cloud-cli.md @@ -40,6 +40,13 @@ Once you install the dbt Cloud CLI, you need to configure it to connect to a dbt 2. Save the `dbt_cloud.yml` file in the `.dbt` directory, which stores your dbt Cloud CLI configuration. Store it in a safe place as it contains API keys. Check out the [FAQs](#faqs) to learn how to create a `.dbt` directory and move the `dbt_cloud.yml` file. + - North America: https://YOUR_ACCESS_URL/cloud-cli + - EMEA: https://emea.dbt.com/cloud-cli + - APAC: https://au.dbt.com/cloud-cli + - North American Cell 1: `https:/ACCOUNT_PREFIX.us1.dbt.com/cloud-cli` + - Single-tenant: `https://YOUR_ACCESS_URL/cloud-cli` + +3. Follow the banner instructions and download the config file to: - Mac or Linux: `~/.dbt/dbt_cloud.yml` - Windows: `C:\Users\yourusername\.dbt\dbt_cloud.yml` @@ -79,7 +86,7 @@ Once you install the dbt Cloud CLI, you need to configure it to connect to a dbt project-id: PROJECT_ID ``` - - To find your project ID, select **Develop** in the dbt Cloud navigation menu. You can use the URL to find the project ID. For example, in `https://cloud.getdbt.com/develop/26228/projects/123456`, the project ID is `123456`. + - To find your project ID, select **Develop** in the dbt Cloud navigation menu. You can use the URL to find the project ID. For example, in `https://YOUR_ACCESS_URL/develop/26228/projects/123456`, the project ID is `123456`. 5. You should now be able to [use the dbt Cloud CLI](#use-the-dbt-cloud-cli) and run [dbt commands](/reference/dbt-commands) like [`dbt environment show`](/reference/commands/dbt-environment) to view your dbt Cloud configuration details or `dbt compile` to compile models in your dbt project. diff --git a/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md b/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md index e2fb122cba3..8fb580ad054 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/develop-in-the-cloud.md @@ -39,7 +39,7 @@ To improve your experience using dbt Cloud, we suggest that you turn off ad bloc ## Prerequisites -- A [dbt Cloud account](https://cloud.getdbt.com/) and [Developer seat license](/docs/cloud/manage-access/seats-and-users) +- A [dbt Cloud account](https://www.getdbt.com/signup) and [Developer seat license](/docs/cloud/manage-access/seats-and-users) - A git repository set up and git provider must have `write` access enabled. See [Connecting your GitHub Account](/docs/cloud/git/connect-github) or [Importing a project by git URL](/docs/cloud/git/import-a-project-by-git-url) for detailed setup instructions - A dbt project connected to a [data platform](/docs/cloud/connect-data-platform/about-connections) - A [development environment and development credentials](#get-started-with-the-cloud-ide) set up diff --git a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md index 138ae4ea3ef..8d80483485c 100644 --- a/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md +++ b/website/docs/docs/cloud/dbt-cloud-ide/ide-user-interface.md @@ -63,18 +63,9 @@ The IDE features some delightful tools and layouts to make it easier for you to - - - - Keep in mind that although you can't delete unmerged local branches in the IDE using this menu, you can [reclone your repository](/docs/cloud/dbt-cloud-ide/ide-user-interface#modals-and-menus), which deletes your local branches and refreshes with the current remote branches, effectively removing the deleted ones. - - - - - Use the **Prune branches** option to remove local branches that have already been deleted from the remote repository. Selecting this triggers a [pop-up modal](#prune-branches-modal), where you can confirm the deletion of the specific local branches, keeping your branch management tidy. Note that this won't delete the branch you're currently on. Pruning branches isn't available for [managed repositories](/docs/collaborate/git/managed-repository) because they don't have a typical remote setup, which prevents remote branch deletion. - - ## Additional editing features - **Minimap —** A Minimap (code outline) gives you a high-level overview of your source code, which is useful for quick navigation and code understanding. A file's minimap is displayed on the upper-right side of the editor. To quickly jump to different sections of your file, click the shaded area. @@ -189,14 +180,10 @@ Use menus and modals to interact with IDE and access useful options to help your The Change Branch modal allows users to switch git branches in the IDE. It can be accessed through the **Change Branch** link or the **Git actions** button under the **Version control** menu. - - - #### Prune branches modal The Prune branches modal allows users to delete local branches that have been deleted from the remote repository, keeping your branch management tidy. This is accessible through the **Git actions** button under the [**Version control** menu](#editing-features). Note that this won't delete the branch you're currently on. Pruning branches isn't available for managed repositories because they don't have a typical remote setup, which prevents remote branch deletion. - - - #### Revert Uncommitted Changes modal The Revert Uncommitted Changes modal is how users revert changes in the IDE. This is accessible via the `Revert File` option above the Version Control Options menu, or via the Git Actions button when there are saved, uncommitted changes in the IDE. diff --git a/website/docs/docs/cloud/manage-access/mfa.md b/website/docs/docs/cloud/manage-access/mfa.md new file mode 100644 index 00000000000..a06251e6468 --- /dev/null +++ b/website/docs/docs/cloud/manage-access/mfa.md @@ -0,0 +1,84 @@ +--- +title: "Multi-factor authentication" +description: "Configure dbt Cloud MFA" +id: "mfa" +sidebar: null +--- + +# Multi-factor authentication + +dbt Cloud provides multiple options for multi-factor authentication (MFA). MFA provides an additional layer of security to username and password logins for Developer and Team plan accounts. The available MFA methods are: + +- SMS verification code (US-based phone numbers only) +- Authenticator app +- Webauthn-compliant security key + +This is available across dbt Cloud plans for users with username and password logins only. + +## Configuration + +You can only have one of the three MFA methods configured per user. These are enabled at the user level, not the account level. + +1. Navigate to the **Account settings** and under **Your profile** click on **Password & Security**. Click **Enroll** next to the preferred method. + + + +Choose the next steps based on your preferred enrollment selection: + + + +2. Select the +1 country code, enter your phone number in the field, and click **Continue**. + + + +3. You will receive an SMS message with a six digit code. Enter the code in dbt Cloud. + + + + + + + +2. Open your preferred authentication app (like Google Authenticator) and scan the QR code. + + + +3. Enter the code provide for "dbt Labs: YOUR_EMAIL_ADDRESS" from your authenticator app into the the field in dbt Cloud. + + + + + +2. Follow the instructions in the modal window and click **Use security key**. + + + +3. Scan the QR code or insert and touch activate your USB key to begin the process. Follow the on-screen prompts. + + + +4. You will be given a backup passcode, store it in a secure location. This key will be useful if the MFA method fails (like a lost or broken phone). + +## Account Recovery + +When setting up MFA, ensure that you store your recovery codes in a secure location, in case your MFA method fails. If you are unable to access your account, reach out to [support@getdbt.com](mailto:support@getdbt.com) for further support. You may need to create a new account if your account cannot be recovered. + +If possible, it's recommended to configure multiple MFA methods so that if one fails, there is a backup option. + +## Disclaimer + +The terms below apply to dbt Cloud’s MFA via SMS program, that dbt Labs (“dbt Labs”, “we”, or “us”) uses to facilitate auto sending of authorization codes to users via SMS for dbt Cloud log-in requests. + +Any clients of dbt Labs that use dbt Cloud Labs 2FA via SMS program (after password is input) are subject to the dbt Labs privacy policy, the client warranty in TOU Section 5.1 second paragraph that Client's use will comply with the Documentation (or similar language in the negotiated service agreement between the parties) and these terms: + +(1) The message frequency is a maximum of 1 message per user login; + +(2) Message and data rates may apply; + +(3) Carriers are not liable for delayed or undelivered messages; + +(4) For help, please reply HELP to the SMS number from which you receive the log-in authorization code(s); + +(5) To opt-out of future SMS messages, please reply STOP to the SMS number from which you receive the log-in authorization code(s). We encourage you to enable an alternate 2FA method before opting-out of SMS messages or you might not be able to log into your account. + +Further questions can be submitted to [support@getdbt.com](mailto:support@getdbt.com). diff --git a/website/docs/docs/cloud/secure/snowflake-privatelink.md b/website/docs/docs/cloud/secure/snowflake-privatelink.md index 7e42275d2fa..6cffc373d3b 100644 --- a/website/docs/docs/cloud/secure/snowflake-privatelink.md +++ b/website/docs/docs/cloud/secure/snowflake-privatelink.md @@ -9,7 +9,7 @@ import SetUpPages from '/snippets/_available-tiers-privatelink.md'; -The following steps will walk you through the setup of a Snowflake AWS PrivateLink endpoint in the dbt Cloud multi-tenant environment. +The following steps walk you through the setup of a Snowflake AWS PrivateLink and Azure Private Link endpoint in the dbt Cloud multi-tenant environment. :::note Snowflake SSO with PrivateLink Users connecting to Snowflake using SSO over a PrivateLink connection from dbt Cloud will also require access to a PrivateLink endpoint from their local workstation. diff --git a/website/docs/docs/collaborate/git/version-control-basics.md b/website/docs/docs/collaborate/git/version-control-basics.md index fd0dfac43d6..edde1bdb48c 100644 --- a/website/docs/docs/collaborate/git/version-control-basics.md +++ b/website/docs/docs/collaborate/git/version-control-basics.md @@ -43,7 +43,7 @@ You can perform git tasks with the git button in the [Cloud IDE](/docs/cloud/dbt | Open pull request | This allows you to open a pull request in Git for peers to review changes before merging into the base branch. | | Pull changes from main | This option is available if you are on any local branch that is behind the remote version of the base branch or the remote version of the branch that you're currently on. | | Pull from remote | This option is available if you’re on the local base branch and changes have recently been pushed to the remote version of the branch. Pulling in changes from the remote repo allows you to pull in the most recent version of the base branch. | -| Rollback to remote | Reset changes to your repository directly from the Cloud IDE. You can rollback your repository back to an earlier clone from your remote. To do this, click on the three dot ellipsis in the bottom right-hand side of the IDE and select **Rollback to remote**. | +| Rollback to remote | Reset changes to your repository directly from the Cloud IDE. You can rollback your repository back to an earlier clone from your remote. To do this, click on the three dot ellipsis in the bottom right-hand side of the IDE and select **Rollback to remote**. | | Refresh git state | This enables you to pull new branches from a different remote branch to your local branch with just one command. | ## Merge conflicts diff --git a/website/docs/docs/collaborate/govern/about-model-governance.md b/website/docs/docs/collaborate/govern/about-model-governance.md index bbc430845d2..a845e941e54 100644 --- a/website/docs/docs/collaborate/govern/about-model-governance.md +++ b/website/docs/docs/collaborate/govern/about-model-governance.md @@ -5,13 +5,6 @@ description: "Information about new features related to model governance" pagination_next: "docs/collaborate/govern/model-access" pagination_prev: null --- - - -:::info New functionality -This functionality is new in v1.5. -::: - - [**Model access**](model-access): Some models are mature, reusable data productions. Others are your team's implementation details on the way there. Mark models as "public" or "private," to make the distinction clear and to control who else can `ref` them. diff --git a/website/docs/docs/collaborate/govern/model-access.md b/website/docs/docs/collaborate/govern/model-access.md index b79e0a4c43f..9fc2ffb021e 100644 --- a/website/docs/docs/collaborate/govern/model-access.md +++ b/website/docs/docs/collaborate/govern/model-access.md @@ -5,12 +5,6 @@ sidebar_label: "Model access" description: "Define model access with group capabilities" --- - - -:::info New functionality -This functionality is new in v1.5 — if you have thoughts, participate in [the discussion on GitHub](https://github.com/dbt-labs/dbt-core/discussions/6730)! -::: - :::info "Model access" is not "User access" **Model groups and access** and **user groups and access** mean two different things. "User groups and access" is a specific term used in dbt Cloud to manage permissions. Refer to [User access](/docs/cloud/manage-access/about-user-access) for more info. @@ -20,9 +14,6 @@ The two concepts will be closely related, as we develop multi-project collaborat - Users in the same dbt Cloud account _without_ access to develop in a project cannot view that project's private models, and they can take a dependency on its public models only. ::: - - - ## Related documentation * [`groups`](/docs/build/groups) * [`access`](/reference/resource-configs/access) @@ -108,8 +99,6 @@ models:
- - Models with `materialized` set to `ephemeral` cannot have the access property set to public. For example, if you have a model config set as: @@ -147,8 +136,6 @@ Parsing Error Node model.jaffle_shop.my_model with 'ephemeral' materialization has an invalid value (public) for the access field ``` - - ## FAQs ### How does model access relate to database permissions? @@ -165,16 +152,6 @@ As we continue to develop multi-project collaboration, `access: public` will mea ### How do I ref a model from another project? - - -In dbt Core v1.5 (and earlier versions), the only way to reference a model from another project is by installing that project as a package, including its full source code. It is not possible to restrict references across projects based on model `access`. - -For more control over per-model access across projects, select v1.6 (or newer) from the version dropdown. - - - - - You can `ref` a model from another project in two ways: 1. [Project dependency](/docs/collaborate/govern/project-dependencies): In dbt Cloud Enterprise, you can use project dependencies to `ref` a model. dbt Cloud uses a behind-the-scenes metadata service to resolve the reference, enabling efficient collaboration across teams and at scale. 2. ["Package" dependency](/docs/build/packages): Another way to `ref` a model from another project is to treat the other project as a package dependency. This requires installing the other project as a package, including its full source code, as well as its upstream dependencies. @@ -203,4 +180,3 @@ restrict-access: True # default is False - diff --git a/website/docs/docs/collaborate/govern/model-contracts.md b/website/docs/docs/collaborate/govern/model-contracts.md index 3f9b9f9c22a..80c4a4f9d81 100644 --- a/website/docs/docs/collaborate/govern/model-contracts.md +++ b/website/docs/docs/collaborate/govern/model-contracts.md @@ -5,15 +5,6 @@ sidebar_label: "Model contracts" description: "Model contracts define a set of parameters validated during transformation" --- - - -:::info New functionality -This functionality is new in v1.5 — if you have thoughts, participate in [the discussion on GitHub](https://github.com/dbt-labs/dbt-core/discussions/6726)! -::: - - - - ## Related documentation * [`contract`](/reference/resource-configs/contract) * [`columns`](/reference/resource-properties/columns) diff --git a/website/docs/docs/collaborate/govern/model-versions.md b/website/docs/docs/collaborate/govern/model-versions.md index 99769012ba6..f255aa9db1a 100644 --- a/website/docs/docs/collaborate/govern/model-versions.md +++ b/website/docs/docs/collaborate/govern/model-versions.md @@ -6,7 +6,7 @@ description: "Version models to help with lifecycle management" keyword: governance, model version, model versioning, dbt model versioning --- - + :::info New functionality This functionality is new in v1.5 — if you have thoughts, participate in [the discussion on GitHub](https://github.com/dbt-labs/dbt-core/discussions/6736)! diff --git a/website/docs/docs/core/source-install.md b/website/docs/docs/core/source-install.md index 0f2e532f36e..06b1c05db49 100644 --- a/website/docs/docs/core/source-install.md +++ b/website/docs/docs/core/source-install.md @@ -40,8 +40,9 @@ python -m pip install -r requirements.txt To install in editable mode, which includes your local changes as you make them: ```shell -python -m pip install -e editable-requirements.txt` instead. +python -m pip install -e editable-requirements.txt` ``` +instead. ### Installing adapter plugins diff --git a/website/docs/docs/dbt-cloud-apis/discovery-use-cases-and-examples.md b/website/docs/docs/dbt-cloud-apis/discovery-use-cases-and-examples.md index edf51cf556c..b01fe6a3b5e 100644 --- a/website/docs/docs/dbt-cloud-apis/discovery-use-cases-and-examples.md +++ b/website/docs/docs/dbt-cloud-apis/discovery-use-cases-and-examples.md @@ -584,8 +584,6 @@ query ($environmentId: BigInt!, $first: Int!) { - - ### How is this model contracted and versioned? To enforce the shape of a model's definition, you can define contracts on models and their columns. You can also specify model versions to keep track of discrete stages in its evolution and use the appropriate one. @@ -628,8 +626,6 @@ query { - - ## Discovery You can use the Discovery API to find and understand relevant datasets and semantic nodes with rich context and metadata. Below are example questions and queries you can run. @@ -908,11 +904,9 @@ Graph example: --> - - ### Which metrics are available? -You can define and query metrics using the [dbt Semantic Layer](/docs/build/about-metricflow), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL). To learn more, refer to [Get started with MetricFlow](/docs/build/sl-getting-started). +You can define and query metrics using the [dbt Semantic Layer](/docs/build/about-metricflow), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL).
Example query @@ -944,10 +938,6 @@ query ($environmentId: BigInt!, $first: Int!) {
-
- - - ## Governance You can use the Discovery API to audit data development and facilitate collaboration within and between teams. @@ -1041,8 +1031,6 @@ query ($environmentId: BigInt!, $first: Int!) { ``` - - ## Development You can use the Discovery API to understand dataset changes and usage and gauge impacts to inform project definition. Below are example questions and queries you can run. diff --git a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md index cb950c4fa8c..1c4d5f387e9 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md +++ b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md @@ -6,14 +6,6 @@ tags: [Semantic Layer, API] hide_table_of_contents: true pagination_next: "docs/dbt-cloud-apis/sl-jdbc" --- - - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - The rapid growth of different tools in the modern data stack has helped data professionals address the diverse needs of different teams. The downside of this growth is the fragmentation of business logic across teams, tools, and workloads.

diff --git a/website/docs/docs/dbt-cloud-apis/sl-graphql.md b/website/docs/docs/dbt-cloud-apis/sl-graphql.md index 421db45ff19..2898b6e5c0a 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-graphql.md +++ b/website/docs/docs/dbt-cloud-apis/sl-graphql.md @@ -5,15 +5,6 @@ description: "Integrate and use the GraphQL API to query your metrics." tags: [Semantic Layer, APIs] --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - - [GraphQL](https://graphql.org/) (GQL) is an open-source query language for APIs. It offers a more efficient and flexible approach compared to traditional RESTful APIs. With GraphQL, users can request specific data using a single query, reducing the need for many server round trips. This improves performance and minimizes network overhead. diff --git a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md index bb5c8fe7918..672f34b2fec 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md +++ b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md @@ -5,14 +5,6 @@ description: "Integrate and use the JDBC API to query your metrics." tags: [Semantic Layer, API] --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer Java Database Connectivity (JDBC) API enables users to query metrics and dimensions using the JDBC protocol, while also providing standard metadata functionality. A JDBC driver is a software component enabling a Java application to interact with a data platform. Here's some more information about our JDBC API: diff --git a/website/docs/docs/dbt-cloud-apis/sl-manifest.md b/website/docs/docs/dbt-cloud-apis/sl-manifest.md index 0dbbeb7b695..e203f4a0754 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-manifest.md +++ b/website/docs/docs/dbt-cloud-apis/sl-manifest.md @@ -7,14 +7,6 @@ sidebar_label: "Semantic manifest" pagination_next: null --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - dbt creates an [artifact](/reference/artifacts/dbt-artifacts) file called the _Semantic Manifest_ (`semantic_manifest.json`), which MetricFlow requires to build and run metric queries properly for the dbt Semantic Layer. This artifact contains comprehensive information about your dbt Semantic Layer. It is an internal file that acts as the integration point with MetricFlow. By using the semantic manifest produced by dbt Core, MetricFlow will instantiate a data flow plan and generate SQL from Semantic Layer query requests. It's a valuable reference that you can use to understand the structure and details of your data models. diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 8afb7e5c391..abaeca3f3d7 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -22,6 +22,8 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - **New**: You can configure metrics at granularities at finer time grains, such as hourly, minute, or even by the second. This is particularly useful for more detailed analysis and for datasets where high-resolution time data is required, such as minute-by-minute event tracking. Refer to [sub-daily granularity](/docs/build/sub-daily-granularity) for more info. ## July 2024 +- **New:** [Connections](/docs/cloud/connect-data-platform/about-connections#connection-management) are now available under **Account settings** as a global setting. Previously, they were found under **Project settings**. This is being rolled out in phases over the coming weeks. +- **New:** Admins can now assign [environment-level permissions](/docs/cloud/manage-access/environment-permissions) to groups for specific roles. - **New:** [Merge jobs](/docs/deploy/merge-jobs) for implementing [continuous deployment (CD)](/docs/deploy/continuous-deployment) workflows are now GA in dbt Cloud. Previously, you had to either set up a custom GitHub action or manually build the changes every time a pull request is merged. - **New**: The ability to lint your SQL files from the dbt Cloud CLI is now available. To learn more, refer to [Lint SQL files](/docs/cloud/configure-cloud-cli#lint-sql-files). - **Behavior change:** dbt Cloud IDE automatically adds a `--limit 100` to preview queries to avoid slow and expensive queries during development. Recently, dbt Core changed how the `limit` is applied to ensure that `order by` clauses are consistently respected. Because of this, queries that already contain a limit clause might now cause errors in the IDE previews. To address this, dbt Labs plans to provide an option soon to disable the limit from being applied. Until then, dbt Labs recommends removing the (duplicate) limit clause from your queries during previews to avoid these IDE errors. diff --git a/website/docs/docs/deploy/deployment-tools.md b/website/docs/docs/deploy/deployment-tools.md index 81c798b7d8c..b41967d463c 100644 --- a/website/docs/docs/deploy/deployment-tools.md +++ b/website/docs/docs/deploy/deployment-tools.md @@ -53,7 +53,7 @@ To use the dbt API to trigger a job in dbt Cloud through ADF: 2. You'll want to create a pipeline in ADF to trigger a dbt Cloud job. 3. Securely fetch the dbt Cloud service token from a key vault in ADF, using a web call as the first step in the pipeline. 4. Set the parameters in the pipeline, including the dbt Cloud account ID and job ID, as well as the name of the key vault and secret that contains the service token. - * You can find the dbt Cloud job and account id in the URL, for example, if your URL is `https://cloud.getdbt.com/deploy/88888/projects/678910/jobs/123456`, the account ID is 88888 and the job ID is 123456 + * You can find the dbt Cloud job and account id in the URL, for example, if your URL is `https://YOUR_ACCESS_URL/deploy/88888/projects/678910/jobs/123456`, the account ID is 88888 and the job ID is 123456 5. Trigger the pipeline in ADF to start the dbt Cloud job and monitor the status of the dbt Cloud job in ADF. 6. In dbt Cloud, you can check the status of the job and how it was triggered in dbt Cloud. diff --git a/website/docs/docs/use-dbt-semantic-layer/dbt-sl.md b/website/docs/docs/use-dbt-semantic-layer/dbt-sl.md index 56204237ae7..73e39589587 100644 --- a/website/docs/docs/use-dbt-semantic-layer/dbt-sl.md +++ b/website/docs/docs/use-dbt-semantic-layer/dbt-sl.md @@ -9,14 +9,6 @@ pagination_next: "guides/sl-snowflake-qs" pagination_prev: null --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer, powered by [MetricFlow](/docs/build/about-metricflow), simplifies the process of defining and using critical business metrics, like `revenue` in the modeling layer (your dbt project). By centralizing metric definitions, data teams can ensure consistent self-service access to these metrics in downstream data tools and applications. The dbt Semantic Layer eliminates duplicate coding by allowing data teams to define metrics on top of existing models and automatically handles data joins. Moving metric definitions out of the BI layer and into the modeling layer allows data teams to feel confident that different business units are working from the same metric definitions, regardless of their tool of choice. If a metric definition changes in dbt, it’s refreshed everywhere it’s invoked and creates consistency across all applications. To ensure secure access control, the dbt Semantic Layer implements robust [access permissions](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) mechanisms. diff --git a/website/docs/docs/use-dbt-semantic-layer/setup-sl.md b/website/docs/docs/use-dbt-semantic-layer/setup-sl.md index 2d4c4c7b8b8..03dc605d83c 100644 --- a/website/docs/docs/use-dbt-semantic-layer/setup-sl.md +++ b/website/docs/docs/use-dbt-semantic-layer/setup-sl.md @@ -8,14 +8,6 @@ tags: [Semantic Layer] With the dbt Semantic Layer, you can centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - ## Prerequisites import SetUp from '/snippets/_v2-sl-prerequisites.md'; diff --git a/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md b/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md index 5f0c50b3e43..d3b79337111 100644 --- a/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md +++ b/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md @@ -7,14 +7,6 @@ tags: [Semantic Layer] pagination_next: null --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer allows you to define metrics and use various interfaces to query them. The Semantic Layer does the heavy lifting to find where the queried data exists in your data platform and generates the SQL to make the request (including performing joins). diff --git a/website/docs/faqs/API/rotate-token.md b/website/docs/faqs/API/rotate-token.md index d21db6f6165..8dea2d0b875 100644 --- a/website/docs/faqs/API/rotate-token.md +++ b/website/docs/faqs/API/rotate-token.md @@ -34,23 +34,21 @@ curl --location --request POST 'https://cloud.getdbt.com/api/v3/accounts/YOUR_AC * Find your `YOUR_CURRENT_TOKEN` by going to **Profile Settings** -> **API Access** and copying the API key. * Find [`YOUR_ACCESS_URL`](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan. - -For example, if `YOUR_ACCOUNT_ID` = `000`, `YOUR_USER_ID` = `123`, `YOUR_CURRENT_PAT_TOKEN` = `abcf9g`, and your `ACCESS_URL` = `cloud.getdbt.com`, then your curl request will be: +If `YOUR_USER_ID` = `123`, `YOUR_CURRENT_TOKEN` = `abcf9g`, then your curl request will be: ``` -curl --location --request POST 'https://cloud.getdbt.com/api/v3/accounts/000/users/123/apikey/' \ +curl --location --request POST 'https://YOUR_ACCESS_URL/api/v2/users/123/apikey/' \ + --header 'Authorization: Token abcf9g' ``` - 2. Find the new key in the API response or in dbt Cloud. 3. To find the new key in dbt Cloud, go to **Profile Settings** -> **API Access**. - ### dbt Cloud deployments -If your [dbt Cloud deployment](/docs/cloud/about-cloud/access-regions-ip-addresses) uses a different access URL, replace `cloud.getdbt.com` with the URL of your instance. +If your [dbt Cloud deployment](/docs/cloud/about-cloud/access-regions-ip-addresses) uses a different access URL, replace `YOUR_ACCESS_URL` with the URL of your instance. For example, if your deployment is Virtual Private dbt: diff --git a/website/docs/faqs/Accounts/find-user-id.md b/website/docs/faqs/Accounts/find-user-id.md index 7f7eca2cbba..09e3ed35a0b 100644 --- a/website/docs/faqs/Accounts/find-user-id.md +++ b/website/docs/faqs/Accounts/find-user-id.md @@ -14,4 +14,4 @@ To find your user ID in dbt Cloud, read the following steps: 3. In the address bar, the number after `/users` is your user ID. 4. Copy that number or save it somewhere safe.
-For example, if the URL is `https://cloud.getdbt.com/settings/accounts/12345/users/67891` — the user ID is `67891`

\ No newline at end of file +For example, if the URL is `https://YOUR_ACCESS_URL/settings/accounts/12345/users/67891` — the user ID is `67891`

\ No newline at end of file diff --git a/website/docs/faqs/Core/install-pip-os-prereqs.md b/website/docs/faqs/Core/install-pip-os-prereqs.md index 1eb6205512a..c8435b44f33 100644 --- a/website/docs/faqs/Core/install-pip-os-prereqs.md +++ b/website/docs/faqs/Core/install-pip-os-prereqs.md @@ -23,13 +23,7 @@ sudo yum install redhat-rpm-config gcc libffi-devel \ ### MacOS - The MacOS requires Python 3.8 or higher to successfully install and run dbt Core. - - - -The MacOS requires Python 3.7 or higher to successfully install and run dbt Core. - To check the Python version: @@ -61,15 +55,6 @@ pip install cryptography~=3.4 Windows requires Python and git to successfully install and run dbt Core. - - Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.8 or higher for Windows](https://www.python.org/downloads/windows/). - - - - - -Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.7 or higher for Windows](https://www.python.org/downloads/windows/). - For further questions, please see the [Python compatibility FAQ](/faqs/Core/install-python-compatibility) diff --git a/website/docs/faqs/Git/gitignore.md b/website/docs/faqs/Git/gitignore.md index 8d966c40e2c..16575861289 100644 --- a/website/docs/faqs/Git/gitignore.md +++ b/website/docs/faqs/Git/gitignore.md @@ -13,7 +13,6 @@ If you encounter issues like problems reverting changes, checking out or creatin To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. - 1. Launch the Cloud IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. @@ -121,6 +120,4 @@ dbt_modules/ - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/faqs/Models/unique-model-names.md b/website/docs/faqs/Models/unique-model-names.md index 7878a5a704c..6d8bd18ac00 100644 --- a/website/docs/faqs/Models/unique-model-names.md +++ b/website/docs/faqs/Models/unique-model-names.md @@ -6,20 +6,8 @@ id: unique-model-names --- - - Within one project: yes! To build dependencies between models, you need to use the `ref` function, and pass in the model name as an argument. dbt uses that model name to uniquely resolve the `ref` to a specific model. As a result, these model names need to be unique, _even if they are in distinct folders_. A model in one project can have the same name as a model in another project (installed as a dependency). dbt uses the project name to uniquely identify each model. We call this "namespacing." If you `ref` a model with a duplicated name, it will resolve to the model within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](/reference/dbt-jinja-functions/ref#ref-project-specific-models) to disambiguate references by specifying the namespace. Those models will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](/docs/build/custom-aliases) and [custom schemas](/docs/build/custom-schemas) for details on how to achieve this. - - - - - -Yes! To build dependencies between models, you need to use the `ref` function, and pass in the model name as an argument. dbt uses that model name to uniquely resolve the `ref` to a specific model. As a result, these model names need to be unique, _even if they are in distinct folders_. - -Often, this question comes up because users want to give two models the same name in their warehouse, splitting them across separate schemas (e.g. `stripe.users` and `app.users`). Checkout the docs on [custom aliases](/docs/build/custom-aliases) and [custom schemas](/docs/build/custom-schemas) to achieve this. - - diff --git a/website/docs/faqs/Project/why-version-2.md b/website/docs/faqs/Project/why-version-2.md index b4e91d6a773..6318ccf037d 100644 --- a/website/docs/faqs/Project/why-version-2.md +++ b/website/docs/faqs/Project/why-version-2.md @@ -6,11 +6,7 @@ id: why-version-2 --- - - Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. Resource yml files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update yml files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future - - diff --git a/website/docs/faqs/Troubleshooting/gitignore.md b/website/docs/faqs/Troubleshooting/gitignore.md index 6ab217ebf07..1519ef99ee3 100644 --- a/website/docs/faqs/Troubleshooting/gitignore.md +++ b/website/docs/faqs/Troubleshooting/gitignore.md @@ -11,8 +11,6 @@ If you can't revert changes, check out a branch, or click commit — this is To fix this, complete the following steps: - - 1. In the dbt Cloud IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: ```bash target/ @@ -37,7 +35,4 @@ dbt_modules/ - - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/faqs/Troubleshooting/job-memory-limits.md b/website/docs/faqs/Troubleshooting/job-memory-limits.md index 377651dae59..06f6a752507 100644 --- a/website/docs/faqs/Troubleshooting/job-memory-limits.md +++ b/website/docs/faqs/Troubleshooting/job-memory-limits.md @@ -15,13 +15,14 @@ Some common reasons for higher memory usage are: ## Resolution -There are various reasons why you could be experiencing this error. We recommend you review your data models to see if there are any opportunities to optimize or refactor them. For example, you can try to reduce the number of columns being selected, use `group` or `where` clauses to filter data early in the query, or use `limit` clauses to reduce the amount of data being processed. +There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. - + If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! diff --git a/website/docs/guides/adapter-creation.md b/website/docs/guides/adapter-creation.md index 4e8479594c2..b6528495260 100644 --- a/website/docs/guides/adapter-creation.md +++ b/website/docs/guides/adapter-creation.md @@ -76,12 +76,12 @@ Differences between databases are encoded into discrete areas: | Components | Code Path | Function | |------------------|---------------------------------------------------|-------------------------------------------------------------------------------| -| Python Classes | `adapters/` | Configuration (See above [Python classes](##python classes) | +| Python classes | `adapters/` | Configuration (Refer to [Python classes](#python classes) | | Macros | `include//macros/adapters/` | SQL API & statement syntax (for example, how to create schema or how to get table info) | | Materializations | `include//macros/materializations/` | Table/view/snapshot/ workflow definitions | -#### Python Classes +#### Python classes These classes implement all the methods responsible for: - Connecting to a database and issuing queries. diff --git a/website/docs/guides/airflow-and-dbt-cloud.md b/website/docs/guides/airflow-and-dbt-cloud.md index 0f1ea0f425f..51ac7668aa9 100644 --- a/website/docs/guides/airflow-and-dbt-cloud.md +++ b/website/docs/guides/airflow-and-dbt-cloud.md @@ -117,7 +117,7 @@ cd airflow-dbt-cloud - Once you hit `save` on the job, make sure you copy the URL and save it for referencing later. The url will look similar to this: ```html -https://cloud.getdbt.com/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ +https://YOUR_ACCESS_URL/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ ``` @@ -134,7 +134,7 @@ Now you have all the working pieces to get up and running with Airflow + dbt Clo ![Connection type](/img/guides/orchestration/airflow-and-dbt-cloud/connection-type.png) -3. Add in your connection details and your default dbt Cloud account id. This is found in your dbt Cloud URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/` +3. Add in your connection details and your default dbt Cloud account id. This is found in your dbt Cloud URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/` ![Connection type](/img/guides/orchestration/airflow-and-dbt-cloud/connection-type-configured.png) @@ -145,7 +145,7 @@ Now you have all the working pieces to get up and running with Airflow + dbt Clo Both IDs are included inside of the dbt Cloud job URL as shown in the following snippets: ```python -# For the dbt Cloud Job URL https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/ +# For the dbt Cloud Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The account_id is 16173 # Update line 28 @@ -153,7 +153,7 @@ default_args={"dbt_cloud_conn_id": "dbt_cloud", "account_id": 16173}, ``` ```python -# For the dbt Cloud Job URL https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/ +# For the dbt Cloud Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The job_id is 65767 # Update line 39 diff --git a/website/docs/guides/bigquery-qs.md b/website/docs/guides/bigquery-qs.md index 1ba5f7b0021..e608efeffc7 100644 --- a/website/docs/guides/bigquery-qs.md +++ b/website/docs/guides/bigquery-qs.md @@ -85,7 +85,7 @@ In order to let dbt connect to your warehouse, you'll need to generate a keyfile 3. Create a service account key for your new project from the [Service accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts?walkthrough_id=iam--create-service-account-keys&start_index=1#step_index=1). For more information, refer to [Create a service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating) in the Google Cloud docs. When downloading the JSON file, make sure to use a filename you can easily remember. For example, `dbt-user-creds.json`. For security reasons, dbt Labs recommends that you protect this JSON file like you would your identity credentials; for example, don't check the JSON file into your version control software. ## Connect dbt Cloud to BigQuery​ -1. Create a new project in [dbt Cloud](https://cloud.getdbt.com/). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. +1. Create a new project in [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. 2. Enter a project name and click **Continue**. 3. For the warehouse, click **BigQuery** then **Next** to set up your connection. 4. Click **Upload a Service Account JSON File** in settings. diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md index 6e130d3a29f..afadf307179 100644 --- a/website/docs/guides/core-to-cloud-1.md +++ b/website/docs/guides/core-to-cloud-1.md @@ -171,9 +171,9 @@ In dbt Cloud, you can set [environment variables](/docs/build/environment-variab In dbt Core, environment variables, or the [`env_var` function](/reference/dbt-jinja-functions/env_var), are defined manually by the developer or within the external application running dbt. ### Environment variables in dbt Cloud - - dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET`). + - dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`). - If your dbt Core environment variables don’t follow this naming convention, perform a [“find and replace”](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#dbt-cloud-ide-features) in your project to make sure all references to these environment variables contain the proper naming conventions. -- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET_``DBT_ENV_SECRET` to obscure them in logs and the UI. +- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET`to obscure them in logs and the UI. diff --git a/website/docs/guides/customize-schema-alias.md b/website/docs/guides/customize-schema-alias.md new file mode 100644 index 00000000000..28d4aada525 --- /dev/null +++ b/website/docs/guides/customize-schema-alias.md @@ -0,0 +1,545 @@ +--- +title: Customize dbt models database, schema, and alias +id: customize-schema-alias +description: "Learn how to properly adjust your generate_schema_name() and generate_alias_name() macros." +displayText: Learn how to adjust your generate schema name and generate alias name. +hoverSnippet: Learn how to adjust your generate schema name and generate alias name. +# time_to_complete: '30 minutes' commenting out until we test +icon: 'guides' +hide_table_of_contents: true +level: 'Advanced' +recently_updated: true +--- + +
+ +## Introduction +This guide explains how to customize the [schema](/docs/build/custom-schemas), [database](/docs/build/custom-databases), and [alias](/docs/build/custom-aliases) naming conventions in dbt to fit your data warehouse governance and design needs. +When we develop dbt models and execute certain [commands](https://docs.getdbt.com/reference/dbt-commands) (such as `dbt run` or `dbt build`), objects (like tables and views) get created in the data warehouse based on these naming conventions. + + + +:::info A word on naming + +Different warehouses have different names for _logical databases_. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. + +::: + + +The following is dbt's out-of-the-box default behavior: + +- The database where the object is created is defined by the database configured at the [environment level in dbt Cloud](/docs/dbt-cloud-environments) or in the [`profiles.yml` file](/docs/core/connect-data-platform/profiles.yml) in dbt Core. + +- The schema depends on whether you have defined a [custom schema](/docs/build/custom-schemas) for the model: + - If you haven't defined a custom schema, dbt creates the object in the default schema. In dbt Cloud this is typically `dbt_username` for development and the default schema for deployment environments. In dbt Core, it uses the schema specified in the `profiles.yml` file. + - If you define a custom schema, dbt concatenates the schema mentioned earlier with the custom one. + - For example, if the configured schema is `dbt_myschema` and the custom one is `marketing`, the objects will be created under `dbt_myschema_marketing`. + - Note that for automated CI jobs, the schema name derives from the job number and PR number: `dbt_cloud_pr__`. + + +- The object name depends on whether an [alias](/reference/resource-configs/alias) has been defined on the model: + - If no alias is defined, the object will be created with the same name as the model, without the `.sql` or `.py` at the end. + - For example, suppose that we have a model where the sql file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and no custom alias is configured. The resulting model will be created in `dbt_myschema_marketing.fct_orders_complete` in the dev environment. + - If an alias is defined, the object will be created with the configured alias. + - For example, suppose that we have a model where the sql file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and the alias is configured to be `fct_orders`. The resulting model will be created in `dbt_myschema_marketing.fct_orders` + +These default rules are a great starting point, and many organizations choose to stick with those without any customization required. + +The defaults allow developers to work in their isolated schemas (sandboxes) without overwriting each other's work — even if they're working on the same tables. + + +## How to customize this behavior + +While the default behavior will fit the needs of most organizations, there are occasions where this approach won't work. + +For example, dbt expects that it has permission to create schemas as needed (and we recommend that the users running dbt have this ability), but it might not be allowed at your company. + +Or, based on how you've designed your warehouse, you may wish to minimize the number of schemas in your dev environment (and avoid schema sprawl by not creating the combination of all developer schemas and custom schemas). + +Alternatively, you may even want your dev schemas to be named after feature branches instead of the developer name. + +For this reason, dbt offers three macros to customize what objects are created in the data warehouse: + +- [`generate_database_name()`](/docs/build/custom-databases#generate_database_name) +- [`generate_schema_name()`](/docs/build/custom-schemas#how-does-dbt-generate-a-models-schema-name) +- [`generate_alias_name()`](/docs/build/custom-aliases#generate_alias_name) + +By overwriting one or multiple of those macros, we can tailor where dbt objects are created in the data warehouse and align with any existing requirement. + + +:::note Key concept + +Models run from two different contexts must result in unique objects in the data warehouse. For example, a developer named Suzie is working on enhancements to `fct_player_stats`, but Darren is developing against the exact same object. + +In order to prevent overwriting each other's work, both Suzie and Darren should each have their unique versions of `fct_player_stats` in the development environment. + +Further, the staging version of `fct_player_stats` should exist in a unique location apart from the development versions, and the production version. + +::: + + +We often leverage the following when customizing these macros: + +- In dbt Cloud, we recommend utilizing [environment variables](/docs/build/environment-variables) to define where the dbt invocation is occurring (dev/stg/prod). + - They can be set at the environment level and all jobs will automatically inherit the default values. We'll add jinja logic (`if/else/endif`) to identify whether the run happens in dev, prod, Ci, and more. + +- Or as an alternative to environment variables, you can use `target.name`. For more information, you can refer to [About target variables](/reference/dbt-jinja-functions/target). + + + + +To allow the database/schema/object name to depend on the current branch, you can use the out of the box `DBT_CLOUD_GIT_BRANCH` environment variable in dbt Cloud [special environment variables](/docs/build/environment-variables#special-environment-variables). + + +## Example use cases + +Here are some typical examples we've encountered with dbt users leveraging those 3 macros and different logic. + + +:::note + +Note that the following examples are not comprehensive and do not cover all the available options. These examples are meant to be templates for you to develop your own behaviors. + +::: + + +- [Use custom schema without concatenating target schema in production](/guides/customize-schema-alias?step=3#1-custom-schemas-without-target-schema-concatenation-in-production) +- [Add developer identities to tables](/guides/customize-schema-alias?step=3#2-static-schemas-add-developer-identities-to-tables) +- [Use branch name as schema prefix](/guides/customize-schema-alias?step=3#3-use-branch-name-as-schema-prefix) +- [Use a static schema for CI](/guides/customize-schema-alias?step=3#4-use-a-static-schema-for-ci) + + +### 1. Custom schemas without target schema concatenation in production + + +The most common use case is using the custom schema without concatenating it with the default schema name when in production. + +To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: + + + + + +```jinja +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- elif env_var('DBT_ENV_TYPE','DEV') == 'PROD' -%} + + {{ custom_schema_name | trim }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Developer 1 | dev | dbt_dev1 |dev.dbt_dev1_marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.dbt_dev2_marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model| +| CI PR 234 | ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model| +| Production | prod | analytics |prod.marketing.my_model | + + +:::note + +We added logic to check if the current dbt run is happening in production or not. This is important, and we explain why in the [What not to do](/guides/customize-schema-alias?step=3#what-not-to-do) section. + +::: + + +### 2. Static schemas: Add developer identities to tables + +Occasionally, we run into instances where the security posture of the organization prevents developers from creating schemas and all developers have to develop in a single schema. + +In this case, we can: + +- Create a new file called generate_schema_name.sql under your macros folder with the following code: + +- Change `generate_schema_name()` to use a single schema for all developers, even if a custom schema is set. +- Update `generate_alias_name()` to append the developer alias and the custom schema to the front of the table name in the dev environment. + - This method is not ideal, as it can cause long table names, but it will let developers see in which schema the model will be created in production. + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- elif env_var('DBT_ENV_TYPE','DEV') != 'CI' -%} + + {{ custom_schema_name | trim }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + + + +```jinja + +{% macro generate_alias_name(custom_alias_name=none, node=none) -%} + + {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} + + {%- if custom_alias_name -%} + + {{ target.schema }}__{{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ target.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} + + {%- else -%} + + {{ target.schema }}__{{ node.name }} + + {%- endif -%} + + {%- else -%} + + {%- if custom_alias_name -%} + + {{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} + + {%- else -%} + + {{ node.name }} + + {%- endif -%} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Developer 1 | dev | dbt_dev1 |dev.marketing.dbt_dev1_my_model | +| Developer 2 | dev | dbt_dev2 |dev.marketing.dbt_dev2_my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model| +| CI PR 234 | ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model| +| Production | prod | analytics |prod.marketing.my_model | + + +### 3. Use branch name as schema prefix + +For teams who prefer to isolate work based on the feature branch, you may want to take advantage of the `DBT_CLOUD_GIT_BRANCH` special environment variable. Please note that developers will write to the exact same schema when they are on the same feature branch. + + +:::note + +The `DBT_CLOUD_GIT_BRANCH` variable is only available within the dbt Cloud IDE and not the Cloud CLI. + +::: + + +We’ve also seen some organizations prefer to organize their dev databases by branch name. This requires implementing similar logic in `generate_database_name()` instead of the `generate_schema_name()` macro. By default, dbt will not automatically create the databases. + +Refer to the [Tips and tricks](https://docs.getdbt.com/guides/customize-schema-alias?step=5) section to learn more. + + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} + + {#- we replace characters not allowed in the schema names by "_" -#} + {%- set re = modules.re -%} + {%- set cleaned_branch = re.sub("\W", "_", env_var('DBT_CLOUD_GIT_BRANCH')) -%} + + {%- if custom_schema_name is none -%} + + {{ cleaned_branch }} + + {%- else -%} + + {{ cleaned_branch }}_{{ custom_schema_name | trim }} + + {%- endif -%} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Branch |Target database| Target schema | Resulting object | +|-------------|:----------:|:-------------:|:-------------:|:---------------------------------:| +| Developer 1 |`featureABC`|dev | dbt_dev1 |dev.featureABC_marketing.my_model | +| Developer 2 |`featureABC`|dev | dbt_dev2 |dev.featureABC_marketing.my_model | +| Developer 1 |`feature123`|dev | dbt_dev1 |dev.feature123_marketing.my_model | +| CI PR 123 | |ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model | +| CI PR 234 | |ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model | +| Production | |prod | analytics |prod.marketing.my_model | + + +When developer 1 and developer 2 are checked out on the same branch, they will generate the same object in the data warehouse. This shouldn't be a problem as being on the same branch means the model's code will be the same for both developers. + + +### 4. Use a static schema for CI + +Some organizations prefer to write their CI jobs to a single schema with the PR identifier prefixed to the front of the table name. It's important to note that this will result in long table names. + +To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: + + + + +```jinja + +{% macro generate_schema_name(custom_schema_name=none, node=none) -%} + + {%- set default_schema = target.schema -%} + + {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} + {# {%- if target.name == 'CI' -%} #} + + {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} + + ci_schema + + {%- elif custom_schema_name is none -%} + + {{ default_schema }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + + + +```jinja + +{% macro generate_alias_name(custom_alias_name=none, node=none) -%} + + {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} + {# {%- if target.name == 'CI' -%} #} + {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} + + {%- if custom_alias_name -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} + + {%- else -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ node.name }} + + {%- endif -%} + + {%- else -%} + + {%- if custom_alias_name -%} + + {{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} + + {%- else -%} + + {{ node.name }} + + {%- endif -%} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:----------------------------------------: | +| Developer 1 | dev | dbt_dev1 |dev.dbt_dev1_marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.dbt_dev2_marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.ci_schema.dbt_pr_123_marketing_my_model | +| CI PR 234 | ci | dbt_pr_234 |ci.ci_schema.dbt_pr_234_marketing_my_model | +| Production | prod | analytics |prod.marketing.my_model | + + +## What not to do + +This section will provide an outline of what users should avoid doing when customizing their schema and alias due to the issues that may arise. + + +### Update generate_schema_name() to always use the custom schema + + +Some people prefer to only use the custom schema when it is set instead of concatenating the default schema with the custom one, as it happens in the out of the box behavior. + + +### Problem + +When modifying the default macro for `generate_schema_name()`, this might result in creating this new version. + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- else -%} + # The following is incorrect as it omits {{ default_schema }} before {{ custom_schema_name | trim }}. + {{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +While it may provide the expected output for production, where a dedicated database is used, it will generate conflicts anywhere people share a database. + + +Let’s look at the example of a model called `my_model` with a custom schema of `marketing`. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Production | prod | analytics |prod.marketing.my_model | +| Developer 1 | dev | dbt_dev1 |dev.marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.marketing.my_model | +| CI PR 234 | ci | dbt_pr_234 |ci.marketing.my_model | + + + +We can see that both developer 1 and developer 2 get the same object for `my_model`. This means that if they both work on this model at the same time, it will be impossible to know if the version currently in the data warehouse is the one from developer 1 and developer 2. + +Similarly, different PRs will result in the exact same object in the data warehouse. If different PRs are open at the same time and modifying the same models, it is very likely that we will get issues, slowing down the whole development and code promotion. + + +### Solution + +As described in the previous example, update the macro to check if dbt is running in production. Only in production should we remove the concatenation and use the custom schema alone. + + +## Tips and tricks + +This section will provide some useful tips on how to properly adjust your `generate_database_name()` and `generate_alias_name()` macros. + + +### Creating non existing databases from dbt + +dbt will automatically try to create a schema if it doesn’t exist and if an object needs to be created in it, but it won’t automatically try to create a database that doesn’t exist. + +So, if your `generate_database_name()` configuration points to different databases, which might not exist, dbt will fail if you do a simple `dbt build`. + +It is still possible to get it working in dbt by creating some macros that will check if a database exists and if not, dbt will create it. You can then call those macros either in [a `dbt run-operation ...` step](/reference/commands/run-operation) in your jobs or as a [`on-run-start` hook](/reference/project-configs/on-run-start-on-run-end). + + +### Assuming context using environment variables rather than `target.name` + + +We prefer to use [environment variables](/docs/build/environment-variables) over `target.name` For a further read, have a look at ([About target variables](/reference/dbt-jinja-functions/target)) to decipher the context of the dbt invocation. + +- `target.name` cannot be set at the environment-level. Therefore, every job within the environment must explicitly specify the `target.name` override. If the job does not have the appropriate `target.name` value set, the database/schema/alias may not resolve properly. Alternatively, environment variable values are inherited by the jobs within their corresponding environment. The environment variable values can also be overwritten within the jobs if needed. + + + + + +- `target.name` requires every developer to input the same value (often ‘dev’) into the target name section of their project development credentials. If a developer doesn’t have the appropriate target name value set, their database/schema/alias may not resolve properly. + + + + + +### Always enforce custom schemas + +Some users prefer to enforce custom schemas on all objects within their projects. This avoids writing to unintended “default” locations. You can add this logic to your `generate_schema_name()` macro to [raise a compilation error](/reference/dbt-jinja-functions/exceptions) if a custom schema is not defined for an object. + + + + +```jinja + + {% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none and node.resource_type == 'model' -%} + + {{ exceptions.raise_compiler_error("Error: No Custom Schema Defined for the model " ~ node.name ) }} + + {%- endif -%} + +``` + + +
diff --git a/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md b/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md index b4cea114f1a..f420b7845a2 100644 --- a/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md +++ b/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md @@ -130,7 +130,7 @@ if __name__ == '__main__': 4. Replace **``** and **``** with the correct values of your environment and [Access URL](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan. - * To find these values, navigate to **dbt Cloud**, select **Deploy -> Jobs**. Select the Job you want to run and copy the URL. For example: `https://cloud.getdbt.com/deploy/000000/projects/111111/jobs/222222` + * To find these values, navigate to **dbt Cloud**, select **Deploy -> Jobs**. Select the Job you want to run and copy the URL. For example: `https://YOUR_ACCESS_URL/deploy/000000/projects/111111/jobs/222222` and therefore valid code would be: Your URL is structured `https:///deploy//projects//jobs/` diff --git a/website/docs/guides/redshift-qs.md b/website/docs/guides/redshift-qs.md index e3685595804..544c18a75d5 100644 --- a/website/docs/guides/redshift-qs.md +++ b/website/docs/guides/redshift-qs.md @@ -166,7 +166,7 @@ Now we are going to load our sample data into the S3 bucket that our Cloudformat select * from stripe.payment; ``` ## Connect dbt Cloud to Redshift -1. Create a new project in [dbt Cloud](https://cloud.getdbt.com/). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. +1. Create a new project in [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. 2. Enter a project name and click **Continue**. 3. For the warehouse, click **Redshift** then **Next** to set up your connection. 4. Enter your Redshift settings. Reference your credentials you saved from the CloudFormation template. diff --git a/website/docs/guides/sl-snowflake-qs.md b/website/docs/guides/sl-snowflake-qs.md index b80f6706c80..d3173744201 100644 --- a/website/docs/guides/sl-snowflake-qs.md +++ b/website/docs/guides/sl-snowflake-qs.md @@ -22,14 +22,6 @@ import ConnectQueryAPI from '/snippets/_sl-connect-and-query-api.md'; import RunProdJob from '/snippets/_sl-run-prod-job.md'; import SlSetUp from '/snippets/_new-sl-setup.md'; - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - ## Introduction The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), powered by [MetricFlow](/docs/build/about-metricflow), simplifies the setup of key business metrics. It centralizes definitions, avoids duplicate code, and ensures easy access to metrics in downstream tools. MetricFlow helps manage company metrics easier, allowing you to define metrics in your dbt project and query them in dbt Cloud with [MetricFlow commands](/docs/build/metricflow-commands). diff --git a/website/docs/guides/starburst-galaxy-qs.md b/website/docs/guides/starburst-galaxy-qs.md index e730abd1fed..3863d80a1b8 100644 --- a/website/docs/guides/starburst-galaxy-qs.md +++ b/website/docs/guides/starburst-galaxy-qs.md @@ -202,7 +202,7 @@ To query the Jaffle Shop data with Starburst Galaxy, you need to create tables u If this role is not listed for you, choose the role you selected in [Connect Starburst Galaxy to the Amazon S3 bucket](#connect-to-s3-bucket) when you added location privilege for your S3 bucket. 3. Click **Clusters** on the left sidebar. 4. Find your cluster in the **View clusters** table and click **Connection info**. Choose **dbt** from the **Select client** dropdown. Keep the **Connection information** modal open. You will use details from that modal in dbt Cloud. -5. In another browser tab, log in to [dbt Cloud](https://cloud.getdbt.com/). +5. In another browser tab, log in to [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). 6. Create a new project in dbt Cloud. From Account settings (using the gear menu in the top right corner), click **+ New Project**. 7. Enter a project name and click **Continue**. 8. Choose **Starburst** as your connection and click **Next**. diff --git a/website/docs/guides/zapier-ms-teams.md b/website/docs/guides/zapier-ms-teams.md index d841ca3305a..171ed19193a 100644 --- a/website/docs/guides/zapier-ms-teams.md +++ b/website/docs/guides/zapier-ms-teams.md @@ -105,7 +105,7 @@ run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Cloud Admin API -url = f'https://cloud.getdbt.com/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' +url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() diff --git a/website/docs/guides/zapier-slack.md b/website/docs/guides/zapier-slack.md index fccce59c948..b1016d4c969 100644 --- a/website/docs/guides/zapier-slack.md +++ b/website/docs/guides/zapier-slack.md @@ -100,7 +100,7 @@ run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Cloud Admin API -url = f'https://cloud.getdbt.com/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' +url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() @@ -260,7 +260,7 @@ api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') commands_to_skip_logs = ['dbt source', 'dbt docs'] run_id = input_data['run_id'] account_id = input_data['account_id'] -url = f'https://cloud.getdbt.com/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' +url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} response = requests.get(url, headers=headers) diff --git a/website/docs/reference/artifacts/other-artifacts.md b/website/docs/reference/artifacts/other-artifacts.md index 75a4653d685..0216acccff0 100644 --- a/website/docs/reference/artifacts/other-artifacts.md +++ b/website/docs/reference/artifacts/other-artifacts.md @@ -25,14 +25,6 @@ Stores the network representation of the dbt resource DAG. ### graph_summary.json - - -:::info New functionality -This functionality is new in v1.6. -::: - - - **Produced by:** [manifest commands](/reference/artifacts/manifest-json) This file is useful for investigating performance issues in dbt Core's graph algorithms. diff --git a/website/docs/reference/artifacts/run-results-json.md b/website/docs/reference/artifacts/run-results-json.md index 75ff3412bfc..ff8da3559fa 100644 --- a/website/docs/reference/artifacts/run-results-json.md +++ b/website/docs/reference/artifacts/run-results-json.md @@ -12,7 +12,8 @@ sidebar_label: "Run results" [`run`](/reference/commands/run) [`seed`](/reference/commands/seed) [`snapshot`](/reference/commands/snapshot) - [`test`](/reference/commands/test) [`run-operation`](/reference/commands/run-operation) + [`test`](/reference/commands/test) + [`run-operation`](/reference/commands/run-operation) This file contains information about a completed invocation of dbt, including timing and status info for each node (model, test, etc) that was executed. In aggregate, many `run_results.json` can be combined to calculate average model runtime, test failure rates, the number of record changes captured by snapshots, etc. diff --git a/website/docs/reference/commands/cmd-docs.md b/website/docs/reference/commands/cmd-docs.md index 60b3049ccf2..cceb8c2ec6e 100644 --- a/website/docs/reference/commands/cmd-docs.md +++ b/website/docs/reference/commands/cmd-docs.md @@ -38,8 +38,6 @@ Use the `--no-compile` argument to skip re-compilation. When this flag is provid dbt docs generate --no-compile ``` - - Use the `--empty-catalog` argument to skip running the database queries to populate `catalog.json`. When this flag is provided, `dbt docs generate` will skip step (3) described above. This is not recommended for production environments, as it means that your documentation will be missing information gleaned from database metadata (the full set of columns in each table, and statistics about those tables). It can speed up `docs generate` in development, when you just want to visualize lineage and other information defined within your project. To learn how to build your documentation in dbt Cloud, refer to [build your docs in dbt Cloud](/docs/collaborate/build-and-view-your-docs). @@ -49,8 +47,6 @@ This is not recommended for production environments, as it means that your docum dbt docs generate --empty-catalog ``` - - ### dbt docs serve This command starts a webserver on port 8080 to serve your documentation locally and opens the documentation site in your default browser. The webserver is rooted in your `target/` directory. Be sure to run `dbt docs generate` before `dbt docs serve` because the `generate` command produces a [catalog metadata artifact](/reference/artifacts/catalog-json) that the `serve` command depends upon. You will see an error message if the catalog is missing. diff --git a/website/docs/reference/commands/compile.md b/website/docs/reference/commands/compile.md index 9422b64b1ea..d67ae80519a 100644 --- a/website/docs/reference/commands/compile.md +++ b/website/docs/reference/commands/compile.md @@ -16,8 +16,6 @@ Some common misconceptions: - `dbt compile` is _not_ a pre-requisite of `dbt run`, or other building commands. Those commands will handle compilation themselves. - If you just want dbt to read and validate your project code, without connecting to the data warehouse, use `dbt parse` instead. - - ### Interactive compile Starting in dbt v1.5, `compile` can be "interactive" in the CLI, by displaying the compiled code of a node or arbitrary dbt-SQL query: @@ -78,8 +76,6 @@ dbt compile --inline "select * from {{ ref('raw_orders') }}" select * from "jaffle_shop"."main"."raw_orders" ``` - - The command accesses the data platform to cache-related metadata, and to run introspective queries. Use the flags: - `--no-populate-cache` to disable the initial cache population. If metadata is needed, it will be a cache miss, requiring dbt to run the metadata query. This is a `dbt` flag, which means you need to add `dbt` as a prefix. For example: `dbt --no-populate-cache`. - `--no-introspect` to disable [introspective queries](/faqs/warehouse/db-connection-dbt-compile#introspective-queries). dbt will raise an error if a model's definition requires running one. This is a `dbt compile` flag, which means you need to add `dbt compile` as a prefix. For example:`dbt compile --no-introspect`. diff --git a/website/docs/reference/commands/debug.md b/website/docs/reference/commands/debug.md index 5f3ee3d5902..7317806f481 100644 --- a/website/docs/reference/commands/debug.md +++ b/website/docs/reference/commands/debug.md @@ -11,16 +11,12 @@ id: "debug" ## Example usage - - Only test the connection to the data platform and skip the other checks `dbt debug` looks for: ```shell $ dbt debug --connection ``` - - Show the configured location for the `profiles.yml` file and exit: ```text diff --git a/website/docs/reference/commands/list.md b/website/docs/reference/commands/list.md index e73699dc78c..9d77d4f348f 100644 --- a/website/docs/reference/commands/list.md +++ b/website/docs/reference/commands/list.md @@ -73,8 +73,6 @@ $ dbt ls --select snowplow.* --output json **Listing JSON output with custom keys** - - ``` $ dbt ls --select snowplow.* --output json --output-keys "name resource_type description" {"name": "snowplow_events", "description": "This is a pretty cool model", ...} @@ -82,10 +80,6 @@ $ dbt ls --select snowplow.* --output json --output-keys "name resource_type des ... ``` - - - - **Listing Semantic models** List all resources upstream of your orders semantic model: @@ -93,8 +87,6 @@ List all resources upstream of your orders semantic model: dbt ls -s +semantic_model:orders ``` - - **Listing file paths** ``` dbt ls --select snowplow.* --output path diff --git a/website/docs/reference/commands/parse.md b/website/docs/reference/commands/parse.md index e709e35aeb5..5e8145762f7 100644 --- a/website/docs/reference/commands/parse.md +++ b/website/docs/reference/commands/parse.md @@ -9,16 +9,12 @@ The `dbt parse` command parses and validates the contents of your dbt project. I It will also produce an artifact with detailed timing information, which is useful to understand parsing times for large projects. Refer to [Project parsing](/reference/parsing) for more information. - - Starting in v1.5, `dbt parse` will write or return a [manifest](/reference/artifacts/manifest-json), enabling you to introspect dbt's understanding of all the resources in your project. By default, the dbt Cloud IDE will attempt a "partial" parse, which means it'll only check changes since the last parse (new or updated parts of your project when you make changes). Since the dbt Cloud IDE automatically parses in the background whenever you save your work, manually running `dbt parse` yourself is likely to be fast because it's just looking at recent changes. As an option, you can tell dbt to check the entire project from scratch by using the `--no-partial-parse` flag. This makes dbt perform a full re-parse of the project, not just the recent changes. - - ``` $ dbt parse 13:02:52 Running with dbt=1.5.0 diff --git a/website/docs/reference/data-test-configs.md b/website/docs/reference/data-test-configs.md index d98392a350b..f47e5bf6979 100644 --- a/website/docs/reference/data-test-configs.md +++ b/website/docs/reference/data-test-configs.md @@ -278,7 +278,11 @@ tests: #### Specify custom configurations for generic data tests -_Currently available in dbt Cloud only. Specifying custom configurations for data tests will become available in dbt Core later this year._ +:::note + +This functionality is supported on versionless dbt Cloud accounts that have opted to ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). Specifying custom configurations for data tests will become available in dbt Core v1.9, available later this year. + +::: Use any custom config key to specify custom configurations for data tests. For example, the following specifies the `snowflake_warehouse` custom config that dbt should use when executing the `accepted_values` data test: @@ -297,4 +301,4 @@ models: ``` -Given the config, the data test runs on a different Snowflake virtual warehouse than the one in your default connection to enable better price-performance with a different warehouse size or more granular cost allocation and visibility. \ No newline at end of file +Given the config, the data test runs on a different Snowflake virtual warehouse than the one in your default connection to enable better price-performance with a different warehouse size or more granular cost allocation and visibility. diff --git a/website/docs/reference/dbt-commands.md b/website/docs/reference/dbt-commands.md index 719c23d65d6..8386cf61731 100644 --- a/website/docs/reference/dbt-commands.md +++ b/website/docs/reference/dbt-commands.md @@ -26,8 +26,6 @@ dbt commands can be `read` or `write` commands: ## Available commands - - The following sections outline the commands supported by dbt and their relevant flags. They are available in all tools and all [supported versions](/docs/dbt-versions/core) unless noted otherwise. You can run these commands in your specific tool by prefixing them with `dbt` — for example, to run the `test` command, type `dbt test`. For information about selecting models on the command line, refer to [Model selection syntax](/reference/node-selection/syntax). @@ -59,58 +57,3 @@ Commands with a ('❌') indicate write commands, commands with a ('✅') indicat | [source](/reference/commands/source) | Provides tools for working with source data (including validating that sources are "fresh") | ✅ | All tools
All [supported versions](/docs/dbt-versions/core) | | [test](/reference/commands/test) | Executes tests defined in a project | ✅ | All tools
All [supported versions](/docs/dbt-versions/core) | Note, use the [`--version`](/reference/commands/version) flag to display the installed dbt Core or dbt Cloud CLI version. (Not applicable for the dbt Cloud IDE). Available on all [supported versions](/docs/dbt-versions/core). -
- - - -Select the tabs that are relevant to your development workflow. For example, if you develop in the dbt Cloud IDE, select **dbt Cloud**. - - - - -Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`. - -- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests) -- [clone](/reference/commands/clone): clone selected nodes from the specified state (requires dbt 1.6 or higher) -- [compile](/reference/commands/compile): compiles (but does not run) the models in a project -- [deps](/reference/commands/deps): downloads dependencies for a project -- [docs](/reference/commands/cmd-docs) : generates documentation for a project -- [retry](/reference/commands/retry): retry the last run `dbt` command from the point of failure (requires dbt 1.6 or later) -- [run](/reference/commands/run): runs the models in a project -- [run-operation](/reference/commands/run-operation): invoke a macro, including running arbitrary maintenance SQL against the database -- [seed](/reference/commands/seed): loads CSV files into the database -- [show](/reference/commands/show): preview table rows post-transformation -- [snapshot](/reference/commands/snapshot): executes "snapshot" jobs defined in a project -- [source](/reference/commands/source): provides tools for working with source data (including validating that sources are "fresh") -- [test](/reference/commands/test): executes tests defined in a project - - - - - -Use the following dbt commands in [dbt Core](/docs/core/installation-overview) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`. - -- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests) -- [clean](/reference/commands/clean): deletes artifacts present in the dbt project -- [clone](/reference/commands/clone): clone selected models from the specified state (requires dbt 1.6 or higher) -- [compile](/reference/commands/compile): compiles (but does not run) the models in a project -- [debug](/reference/commands/debug): debugs dbt connections and projects -- [deps](/reference/commands/deps): downloads dependencies for a project -- [docs](/reference/commands/cmd-docs) : generates documentation for a project -- [init](/reference/commands/init): initializes a new dbt project -- [list](/reference/commands/list): lists resources defined in a dbt project -- [parse](/reference/commands/parse): parses a project and writes detailed timing info -- [retry](/reference/commands/retry): retry the last run `dbt` command from the point of failure (requires dbt 1.6 or higher) -- [rpc](/reference/commands/rpc): runs an RPC server that clients can submit queries to -- [run](/reference/commands/run): runs the models in a project -- [run-operation](/reference/commands/run-operation): invoke a macro, including running arbitrary maintenance SQL against the database -- [seed](/reference/commands/seed): loads CSV files into the database -- [show](/reference/commands/show): preview table rows post-transformation -- [snapshot](/reference/commands/snapshot): executes "snapshot" jobs defined in a project -- [source](/reference/commands/source): provides tools for working with source data (including validating that sources are "fresh") -- [test](/reference/commands/test): executes tests defined in a project - - - - - diff --git a/website/docs/reference/dbt-jinja-functions/builtins.md b/website/docs/reference/dbt-jinja-functions/builtins.md index d3241ab849a..fa5f57a48e0 100644 --- a/website/docs/reference/dbt-jinja-functions/builtins.md +++ b/website/docs/reference/dbt-jinja-functions/builtins.md @@ -21,8 +21,6 @@ The `builtins` variable is a dictionary containing the following keys: Using the `builtins` variable in this way is an advanced development workflow. Users should be ready to maintain and update these overrides when upgrading in the future. ::: - - From dbt v1.5 and higher, use the following macro to override the `ref` method available in the model compilation context to return a [Relation](/reference/dbt-classes#relation) with the database name overriden to `dev`. It includes logic to extract user-provided arguments, including version, and call the builtins.ref() function with either a single modelname argument or both packagename and modelname arguments, based on the number of positional arguments in varargs. @@ -59,7 +57,6 @@ Note that the `ref`, `source`, and `config` functions can't be overridden with a {% endmacro %} ``` - Logic within the ref macro can also be used to control which elements of the model path are rendered when run, for example the following logic renders only the schema and object identifier, but not the database reference i.e. `my_schema.my_model` rather than `my_database.my_schema.my_model`. This is especially useful when using snowflake as a warehouse, if you intend to change the name of the database post-build and wish the references to remain accurate. diff --git a/website/docs/reference/dbt-jinja-functions/env_var.md b/website/docs/reference/dbt-jinja-functions/env_var.md index a239ce4ff13..28feccc30e4 100644 --- a/website/docs/reference/dbt-jinja-functions/env_var.md +++ b/website/docs/reference/dbt-jinja-functions/env_var.md @@ -58,7 +58,7 @@ models: ### Secrets -For certain configurations, you can use "secret" env vars. Any env var named with the prefix `DBT_ENV_SECRET_``DBT_ENV_SECRET` will be: +For certain configurations, you can use "secret" env vars. Any env var named with the prefix `DBT_ENV_SECRET` will be: - Available for use in `profiles.yml` + `packages.yml`, via the same `env_var()` function - Disallowed everywhere else, including `dbt_project.yml` and model SQL, to prevent accidentally writing these secret values to the or metadata artifacts - Scrubbed from dbt logs and replaced with `*****`, any time its value appears in those logs (even if the env var was not called directly) @@ -98,5 +98,5 @@ select 1 as id ### dbt Cloud usage -If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET`). Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. +If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`). Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. diff --git a/website/docs/reference/dbt-jinja-functions/flags.md b/website/docs/reference/dbt-jinja-functions/flags.md index 534a0fa8987..9df6f9a58ea 100644 --- a/website/docs/reference/dbt-jinja-functions/flags.md +++ b/website/docs/reference/dbt-jinja-functions/flags.md @@ -48,28 +48,6 @@ select 1 as id - - -```shell -$ DBT_ENV_CUSTOM_ENV_MYVAR=myvalue dbt compile -s my_model -``` - - - -```sql --- invocation_args_dict: --- {'write_json': True, 'use_colors': True, 'printer_width': 80, 'version_check': True, 'partial_parse': True, 'static_parser': True, 'profiles_dir': '/Users/.../.dbt', 'send_anonymous_usage_stats': False, 'event_buffer_size': 100000, 'quiet': False, 'no_print': False, 'parse_only': False, 'which': 'compile', 'rpc_method': 'compile', 'indirect_selection': 'eager'} - --- dbt_metadata_envs: --- {'MYVAR': 'myvalue'} - -select 1 as id -``` - - - - - The `invocation_command` key within `invocation_args_dict` includes the entire subcommand when it compiles: ```shell @@ -89,8 +67,4 @@ $ DBT_ENV_CUSTOM_ENV_MYVAR=myvalue dbt compile -s my_model -- {'MYVAR': 'myvalue'} select 1 as id -``` - - - - +``` \ No newline at end of file diff --git a/website/docs/reference/dbt-jinja-functions/graph.md b/website/docs/reference/dbt-jinja-functions/graph.md index bea09c326e8..ccb05da8f2b 100644 --- a/website/docs/reference/dbt-jinja-functions/graph.md +++ b/website/docs/reference/dbt-jinja-functions/graph.md @@ -23,8 +23,6 @@ to understand how to effectively use this variable. The `graph` context variable is a dictionary which maps node ids onto dictionary representations of those nodes. A simplified example might look like: - - ```json { "nodes": { @@ -81,8 +79,6 @@ representations of those nodes. A simplified example might look like: } ``` - - The exact contract for these model and source nodes is not currently documented, but that will change in the future. diff --git a/website/docs/reference/dbt-jinja-functions/ref.md b/website/docs/reference/dbt-jinja-functions/ref.md index 8212359c1f1..535093ead27 100644 --- a/website/docs/reference/dbt-jinja-functions/ref.md +++ b/website/docs/reference/dbt-jinja-functions/ref.md @@ -94,12 +94,8 @@ select * from {{ ref('project_or_package', 'model_name') }} We recommend using two-argument `ref` any time you are referencing a model defined in a different package or project. While not required in all cases, it's more explicit for you, for dbt, and future readers of your code. - - We especially recommend using two-argument `ref` to avoid ambiguity, in cases where a model name is duplicated across multiple projects or installed packages. If you use one-argument `ref` (just the `model_name`), dbt will look for a model by that name in the same namespace (package or project); if it finds none, it will raise an error. - - **Note:** The `project_or_package` should match the `name` of the project/package, as defined in its `dbt_project.yml`. This might be different from the name of the repository. It never includes the repository's organization name. For example, if you use the [`fivetran/stripe`](https://hub.getdbt.com/fivetran/stripe/latest/) package, the package name is `stripe`, not `fivetran/stripe`. ### Forcing Dependencies diff --git a/website/docs/reference/dbt_project.yml.md b/website/docs/reference/dbt_project.yml.md index 6166be1df6d..08261dd6932 100644 --- a/website/docs/reference/dbt_project.yml.md +++ b/website/docs/reference/dbt_project.yml.md @@ -3,13 +3,10 @@ Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — thi - dbt uses [YAML](https://yaml.org/) in a few different places. If you're new to YAML, it would be worth learning how arrays, dictionaries, and strings are represented. - - - By default, dbt looks for the `dbt_project.yml` in your current working directory and its parents, but you can set a different directory using the `--project-dir` flag or the `DBT_PROJECT_DIR` environment variable. -- Specify your dbt Cloud project ID in the `dbt_project.yml` file using `project-id` under the `dbt-cloud` config. Find your project ID in your dbt Cloud project URL: For example, in `https://cloud.getdbt.com/11/projects/123456`, the project ID is `123456`. +- Specify your dbt Cloud project ID in the `dbt_project.yml` file using `project-id` under the `dbt-cloud` config. Find your project ID in your dbt Cloud project URL: For example, in `https://YOUR_ACCESS_URL/11/projects/123456`, the project ID is `123456`. - - Note, you can't set up a "property" in the `dbt_project.yml` file if it's not a config (an example is [macros](/reference/macro-properties)). This applies to all types of resources. Refer to [Configs and properties](/reference/configs-and-properties) for more detail. @@ -99,7 +96,7 @@ vars:
- + @@ -169,76 +166,6 @@ vars: - - - - - -```yml -[name](/reference/project-configs/name): string - -[config-version](/reference/project-configs/config-version): 2 -[version](/reference/project-configs/version): version - -[profile](/reference/project-configs/profile): profilename - -[model-paths](/reference/project-configs/model-paths): [directorypath] -[seed-paths](/reference/project-configs/seed-paths): [directorypath] -[test-paths](/reference/project-configs/test-paths): [directorypath] -[analysis-paths](/reference/project-configs/analysis-paths): [directorypath] -[macro-paths](/reference/project-configs/macro-paths): [directorypath] -[snapshot-paths](/reference/project-configs/snapshot-paths): [directorypath] -[docs-paths](/reference/project-configs/docs-paths): [directorypath] -[asset-paths](/reference/project-configs/asset-paths): [directorypath] - -[target-path](/reference/global-configs/json-artifacts): directorypath -[log-path](/reference/global-configs/logs): directorypath -[packages-install-path](/reference/project-configs/packages-install-path): directorypath - -[clean-targets](/reference/project-configs/clean-targets): [directorypath] - -[query-comment](/reference/project-configs/query-comment): string - -[require-dbt-version](/reference/project-configs/require-dbt-version): version-range | [version-range] - -[quoting](/reference/project-configs/quoting): - database: true | false - schema: true | false - identifier: true | false - -models: - [](/reference/model-configs) - -seeds: - [](/reference/seed-configs) - -snapshots: - [](/reference/snapshot-configs) - -sources: - [](source-configs) - -tests: - [](/reference/data-test-configs) - -vars: - [](/docs/build/project-variables) - -[on-run-start](/reference/project-configs/on-run-start-on-run-end): sql-statement | [sql-statement] -[on-run-end](/reference/project-configs/on-run-start-on-run-end): sql-statement | [sql-statement] - -[dispatch](/reference/project-configs/dispatch-config): - - macro_namespace: packagename - search_order: [packagename] - -[restrict-access](/docs/collaborate/govern/model-access): true | false - -``` - - - - - ## Naming convention It's important to follow the correct YAML naming conventions for the configs in your `dbt_project.yml` file to ensure dbt can process them properly. This is especially true for resource types with more than one word. diff --git a/website/docs/reference/events-logging.md b/website/docs/reference/events-logging.md index de79d8d9171..f78774e6310 100644 --- a/website/docs/reference/events-logging.md +++ b/website/docs/reference/events-logging.md @@ -65,7 +65,7 @@ Many events are fired while compiling or running a specific DAG node (model, see | `node_finished_at` | Timestamp when node processing completed | | `node_name` | Name of this model/seed/test/etc | | `node_path` | File path to where this resource is defined | -| `node_relation` | Nested object containing this node's database representation: `database`, `schema`, `alias`, and full `relation_name` with quoting & inclusion policies applied | +| `node_relation` | Nested object containing this node's database representation: `database`, `schema`, `alias`, and full `relation_name` with quoting & inclusion policies applied | | `node_started_at` | Timestamp when node processing started | | `node_status` | Current status of the node, either `RunningStatus` (while running) or `NodeStatus` (finished) as defined in [the result contract](https://github.com/dbt-labs/dbt-core/blob/eba90863ed4043957330ea44ca267db1a2d81fcd/core/dbt/contracts/results.py#L75-L88) | | `resource_type` | `model`, `test`, `seed`, `snapshot`, etc. | @@ -121,10 +121,7 @@ Many events are fired while compiling or running a specific DAG node (model, see Older versions of `dbt-core` made available a full history of events fired during an invocation, in the form of an `EVENT_HISTORY` object. - - When [invoking dbt programmatically](programmatic-invocations#registering-callbacks), it is possible to register a callback on dbt's `EventManager`. This allows access to structured events as Python objects, to enable custom logging and integration with other systems. - The Python interface into events is significantly less mature than the structured logging interface. For all standard use cases, we recommend parsing JSON-formatted logs. diff --git a/website/docs/reference/global-configs/cache.md b/website/docs/reference/global-configs/cache.md index 7687df30339..1a74fef8d30 100644 --- a/website/docs/reference/global-configs/cache.md +++ b/website/docs/reference/global-configs/cache.md @@ -4,8 +4,6 @@ id: "cache" sidebar: "Cache" --- - - ### Cache population At the start of runs, dbt caches metadata about all the objects in all the schemas where it might materialize resources (such as models). By default, dbt populates the cache with information on all schemas related to the project. @@ -28,6 +26,3 @@ Or, to improve speed and performance while focused on developing Salesforce mode dbt --cache-selected-only run --select salesforce ``` - - - diff --git a/website/docs/reference/global-configs/indirect-selection.md b/website/docs/reference/global-configs/indirect-selection.md index 07116d4c4b1..729176a1ff4 100644 --- a/website/docs/reference/global-configs/indirect-selection.md +++ b/website/docs/reference/global-configs/indirect-selection.md @@ -16,8 +16,6 @@ When all flags are set, the order of precedence is as follows. Refer to [About g You can set the flag to: `empty`, `buildable`, `cautious`, or `eager` (default). By default, dbt indirectly selects all tests if they touch any resource you select. Learn more about these options in [Indirect selection in Test selection examples](/reference/node-selection/test-selection-examples?indirect-selection-mode=eager#indirect-selection). - - The following is a visualization of the impact `--indirect-selection` and the various flags have using three models, three tests, and `dbt build` as an example: @@ -36,8 +34,6 @@ The following is a visualization of the impact `--indirect-selection` and the va - - For example, you can run tests that only refer to selected nodes using a CLI configuration: diff --git a/website/docs/reference/global-configs/logs.md b/website/docs/reference/global-configs/logs.md index 19ca8da6b5c..beca0ded49e 100644 --- a/website/docs/reference/global-configs/logs.md +++ b/website/docs/reference/global-configs/logs.md @@ -8,12 +8,8 @@ sidebar: "logs" dbt outputs logs to two different locations: CLI console and the log file. - - The `LOG_FORMAT` and `LOG_FORMAT_FILE` configs specify how dbt's logs should be formatted, and they each have the same options: `json`, `text`, and `debug`. - - ```text @@ -44,8 +40,6 @@ The `json` format outputs fully structured logs in the format {"data": {"adapter_name": "postgres", "adapter_version": "=1.8.0"}, "info": {"category": "", "code": "E034", "extra": {}, "invocation_id": "82131fa0-d2b4-4a77-9436-019834e22746", "level": "info", "msg": "Registered adapter: postgres=1.8.0", "name": "AdapterRegistered", "pid": 7875, "thread": "MainThread", "ts": "2024-05-29T23:32:56.437986Z"}} ``` - - When the `LOG_FORMAT` is set explicitly, it will take affect in both the console and log files whereas the `LOG_FORMAT_FILE` only affects the log file. @@ -56,8 +50,6 @@ dbt --log-format-file json run - - :::tip Tip: verbose structured logs Use `json` formatting value in conjunction with the `DEBUG` config to produce rich log information which can be piped into monitoring tools for analysis: @@ -70,8 +62,6 @@ See [structured logging](/reference/events-logging#structured-logging) for more ::: - - ### Log Level The `LOG_LEVEL` config sets the minimum severity of events captured in the console and file logs. This is a more flexible alternative to the `--debug` flag. The available options for the log levels are `debug`, `info`, `warn`, `error`, or `none`. @@ -90,9 +80,6 @@ To set the file log level as a different value than the console, use the `--log- dbt --log-level-file error run ``` - - - ### Debug-level logging The `DEBUG` config redirects dbt's debug logs to standard output. This has the effect of showing debug-level log information in the terminal in addition to the `logs/dbt.log` file. This output is verbose. @@ -156,8 +143,6 @@ The `LOG_CACHE_EVENTS` config allows detailed logging for [relational cache](ref dbt --log-cache-events compile ``` - - ### Color You can set the color preferences for the file logs only within `profiles.yml` or using the `--use-colors-file / --no-use-colors-file` flags. @@ -175,5 +160,3 @@ config: dbt --use-colors-file run dbt --no-use-colors-file run ``` - - diff --git a/website/docs/reference/global-configs/print-output.md b/website/docs/reference/global-configs/print-output.md index fc129b162a7..7675dbb82c2 100644 --- a/website/docs/reference/global-configs/print-output.md +++ b/website/docs/reference/global-configs/print-output.md @@ -6,8 +6,6 @@ sidebar: "Print output" ### Suppress `print()` messages in stdout - - By default, dbt includes [`print()`](/reference/dbt-jinja-functions/print) messages in standard out (stdout). You can use the `DBT_PRINT` environment variable to prevent these messages from showing up in stdout. :::warning Syntax deprecation @@ -16,8 +14,6 @@ The original `DBT_NO_PRINT` environment variable has been deprecated, starting w ::: - - Supply `--no-print` flag to `dbt run` to suppress `print()` messages from showing in stdout. ```text @@ -54,7 +50,6 @@ config: dbt --use-colors run dbt --no-use-colors run ``` - You can set the color preferences for the file logs only within `profiles.yml` or using the `--use-colors-file / --no-use-colors-file` flags. @@ -71,5 +66,3 @@ config: dbt --use-colors-file run dbt --no-use-colors-file run ``` - - diff --git a/website/docs/reference/node-selection/defer.md b/website/docs/reference/node-selection/defer.md index bbcc5f7d567..27cc3dd78c5 100644 --- a/website/docs/reference/node-selection/defer.md +++ b/website/docs/reference/node-selection/defer.md @@ -8,12 +8,8 @@ Defer requires that a manifest from a previous dbt invocation be passed to the ` An alternative command that accomplishes similar functionality for different use cases is `dbt clone` - see the docs for [clone](/reference/commands/clone#when-to-use-dbt-clone-instead-of-deferral) for more information. - - It is possible to use separate state for `state:modified` and `--defer`, by passing paths to different manifests to each of the `--state`/`DBT_STATE` and `--defer-state`/`DBT_DEFER_STATE`. This enables more granular control in cases where you want to compare against logical state from one environment or past point in time, and defer to applied state from a different environment or point in time. If `--defer-state` is not specified, deferral will use the manifest supplied to `--state`. In most cases, you will want to use the same state for both: compare logical changes against production, and also "fail over" to the production environment for unbuilt upstream resources. - - ### Usage ```shell @@ -43,11 +39,8 @@ When using defer, you may be selecting from production datasets, development dat - if you apply env-specific limits in dev but not prod, as you may end up selecting more data than you expect - when executing tests that depend on multiple parents (e.g. `relationships`), since you're testing "across" environments - - Deferral requires both `--defer` and `--state` to be set, either by passing flags explicitly or by setting environment variables (`DBT_DEFER` and `DBT_STATE`). If you use dbt Cloud, read about [how to set up CI jobs](/docs/deploy/continuous-integration). - #### Favor state diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index 438847c6d5b..37f50f734e7 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -6,8 +6,6 @@ Selector methods return all resources that share a common property, using the syntax `method:value`. While it is recommended to explicitly denote the method, you can omit it (the default value will be one of `path`, `file` or `fqn`). - - Many of the methods below support Unix-style wildcards: @@ -24,8 +22,6 @@ dbt list --select "*.folder_name.*" dbt list --select "package:*_source" ``` - - ### The "tag" method The `tag:` method is used to select models that match a specified [tag](/reference/resource-configs/tags). @@ -199,12 +195,8 @@ dbt test --select "test_name:range_min_max" # run all instances of a custom **N.B.** State-based selection is a powerful, complex feature. Read about [known caveats and limitations](/reference/node-selection/state-comparison-caveats) to state comparison. - - The `state` method is used to select nodes by comparing them against a previous version of the same project, which is represented by a [manifest](/reference/artifacts/manifest-json). The file path of the comparison manifest _must_ be specified via the `--state` flag or `DBT_STATE` environment variable. - - `state:new`: There is no node with the same `unique_id` in the comparison manifest `state:modified`: All new nodes, plus any changes to existing nodes. @@ -227,16 +219,12 @@ Because state comparison is complex, and everyone's project is different, dbt su Remember that `state:modified` includes _all_ of the criteria above, as well as some extra resource-specific criteria, such as modifying a source's `freshness` or `quoting` rules or an exposure's `maturity` property. (View the source code for the full set of checks used when comparing [sources](https://github.com/dbt-labs/dbt-core/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L660-L681), [exposures](https://github.com/dbt-labs/dbt-core/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L768-L783), and [executable nodes](https://github.com/dbt-labs/dbt-core/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L319-L330).) - - There are two additional `state` selectors that complement `state:new` and `state:modified` by representing the inverse of those functions: - `state:old` — A node with the same `unique_id` exists in the comparison manifest - `state:unmodified` — All existing nodes with no changes These selectors can help you shorten run times by excluding unchanged nodes. Currently, no subselectors are available at this time, but that might change as use cases evolve. - - ### The "exposure" method The `exposure` method is used to select parent resources of a specified [exposure](/docs/build/exposures). Use in conjunction with the `+` operator. @@ -279,7 +267,6 @@ The following dbt commands produce `sources.json` artifacts whose results can be After issuing one of the above commands, you can reference the source freshness results by adding a selector to a subsequent command as follows: - ```bash # You can also set the DBT_STATE environment variable instead of the --state flag. @@ -287,13 +274,8 @@ dbt source freshness # must be run again to compare current to previous state dbt build --select "source_status:fresher+" --state path/to/prod/artifacts ``` - - - ### The "group" method - - The `group` method is used to select models defined within a [group](/reference/resource-configs/group). @@ -301,12 +283,8 @@ The `group` method is used to select models defined within a [group](/reference/ dbt run --select "group:finance" # run all models that belong to the finance group. ``` - - ### The "access" method - - The `access` method selects models based on their [access](/reference/resource-configs/access) property. ```bash @@ -315,12 +293,8 @@ dbt list --select "access:private" # list all private models dbt list --select "access:protected" # list all protected models ``` - - ### The "version" method - - The `version` method selects [versioned models](/docs/collaborate/govern/model-versions) based on their [version identifier](/reference/resource-properties/versions) and [latest version](/reference/resource-properties/latest_version). ```bash @@ -331,13 +305,7 @@ dbt list --select "version:old" # versions older than the 'latest' versi dbt list --select "version:none" # models that are *not* versioned ``` - - ### The "semantic_model" method - -Supported in v1.6 or newer. - - The `semantic_model` method selects [semantic models](/docs/build/semantic-models). @@ -346,8 +314,6 @@ dbt list --select "semantic_model:*" # list all semantic models dbt list --select "+semantic_model:orders" # list your semantic model named "orders" and all upstream resources ``` - - ### The "saved_query" method Supported in v1.7 or newer. diff --git a/website/docs/reference/node-selection/syntax.md b/website/docs/reference/node-selection/syntax.md index ce7d27cb6b6..a46c4145217 100644 --- a/website/docs/reference/node-selection/syntax.md +++ b/website/docs/reference/node-selection/syntax.md @@ -136,8 +136,6 @@ Together, the `state:` selector and deferral enable ["slim CI"](/best-practices/ State and defer can be set by environment variables as well as CLI flags: - - - `--state` or `DBT_STATE`: file path - `--defer` or `DBT_DEFER`: boolean @@ -147,18 +145,12 @@ In dbt v1.5, we deprecated the original syntax for state (`DBT_ARTIFACT_STATE_PA ::: - - - - - `--state` or `DBT_STATE`: file path - `--defer` or `DBT_DEFER`: boolean - `--defer-state` or `DBT_DEFER_STATE`: file path to use for deferral only (optional) If `--defer-state` is not specified, deferral will use the artifacts supplied by `--state`. This enables more granular control in cases where you want to compare against logical state from one environment or past point in time, and defer to applied state from a different environment or point in time. - - If both the flag and env var are provided, the flag takes precedence. #### Notes: diff --git a/website/docs/reference/node-selection/test-selection-examples.md b/website/docs/reference/node-selection/test-selection-examples.md index 11362b2364b..445da893ce5 100644 --- a/website/docs/reference/node-selection/test-selection-examples.md +++ b/website/docs/reference/node-selection/test-selection-examples.md @@ -35,19 +35,13 @@ In both cases, `test_type` checks a property of the test itself. These are forms ### Indirect selection - - - - ### Indirect selection examples - - To visualize these methods, suppose you have `model_a`, `model_b`, and `model_c` and associated data tests. The following illustrates which tests will be run when you execute `dbt build` with the various indirect selection modes: @@ -115,8 +109,6 @@ dbt build --select "orders" --indirect-selection=empty - - ### Test selection syntax examples diff --git a/website/docs/reference/node-selection/yaml-selectors.md b/website/docs/reference/node-selection/yaml-selectors.md index d911eb44baa..ff6628919b7 100644 --- a/website/docs/reference/node-selection/yaml-selectors.md +++ b/website/docs/reference/node-selection/yaml-selectors.md @@ -57,8 +57,6 @@ This is the most thorough syntax, which can include the operator-equivalent keyw Review [methods](/reference/node-selection/methods) for the available list. - - ```yml definition: method: tag @@ -77,9 +75,6 @@ definition: indirect_selection: eager | cautious | buildable | empty # include all tests selected indirectly? eager by default ``` - - - The `*` operator to select all nodes can be written as: ```yml definition: @@ -113,16 +108,11 @@ Note: The `exclude` argument in YAML selectors is subtly different from the `--exclude` CLI argument. Here, `exclude` _always_ returns a [set difference](https://en.wikipedia.org/wiki/Complement_(set_theory)), and it is always applied _last_ within its scope. - - When more than one "yeslist" (`--select`) is passed, they are treated as a [union](/reference/node-selection/set-operators#unions) rather than an [intersection](/reference/node-selection/set-operators#intersections). Same thing when there is more than one "nolist" (`--exclude`). - #### Indirect selection - - As a general rule, dbt will indirectly select _all_ tests if they touch _any_ resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious", "buildable", or "empty" by setting `indirect_selection` for a specific criterion: ```yml @@ -145,8 +135,6 @@ As a general rule, dbt will indirectly select _all_ tests if they touch _any_ re If provided, a YAML selector's `indirect_selection` value will take precedence over the CLI flag `--indirect-selection`. Because `indirect_selection` is defined separately for _each_ selection criterion, it's possible to mix eager/cautious/buildable/empty modes within the same definition, to achieve the exact behavior that you need. Remember that you can always test out your critiera with `dbt ls --selector`. - - See [test selection examples](/reference/node-selection/test-selection-examples) for more details about indirect selection. ## Example diff --git a/website/docs/reference/project-configs/config-version.md b/website/docs/reference/project-configs/config-version.md index c785b2d4c22..78b9be49a20 100644 --- a/website/docs/reference/project-configs/config-version.md +++ b/website/docs/reference/project-configs/config-version.md @@ -3,11 +3,7 @@ datatype: integer description: "Read this guide to understand the config-version configuration in dbt." --- - - -Starting in dbt v1.5, the `config-version:` tag is optional. - - +The `config-version:` tag is optional. @@ -18,9 +14,9 @@ config-version: 2 ## Definition -Specify your `dbt_project.yml` as using the v2 structure. - This configuration is optional. +Specify your `dbt_project.yml` as using the v2 structure. ## Default + Without this configuration, dbt will assume your `dbt_project.yml` uses the version 2 syntax. Version 1 was deprecated in dbt v0.19.0. diff --git a/website/docs/reference/resource-configs/bigquery-configs.md b/website/docs/reference/resource-configs/bigquery-configs.md index 5a95c8f9567..851a93c2bcc 100644 --- a/website/docs/reference/resource-configs/bigquery-configs.md +++ b/website/docs/reference/resource-configs/bigquery-configs.md @@ -336,7 +336,7 @@ dbt supports the specification of BigQuery labels for the tables and BigQuery key-value pair entries for labels larger than 63 characters are truncated. + BigQuery key-value pair entries for labels larger than 63 characters are truncated. **Configuring labels in a model file** diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index fc2b2c81828..68224802026 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -7,20 +7,7 @@ id: "databricks-configs" When materializing a model as `table`, you may include several optional configs that are specific to the dbt-databricks plugin, in addition to the standard [model configs](/reference/model-configs). - - -| Option | Description | Required? | Example | -|---------------------|------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|--------------------------| -| file_format | The file format to use when creating tables (`parquet`, `delta`, `hudi`, `csv`, `json`, `text`, `jdbc`, `orc`, `hive` or `libsvm`). | Optional | `delta` | -| location_root | The created table uses the specified directory to store its data. The table alias is appended to it. | Optional | `/mnt/root` | -| partition_by | Partition the created table by the specified columns. A directory is created for each partition. | Optional | `date_day` | -| clustered_by | Each partition in the created table will be split into a fixed number of buckets by the specified columns. | Optional | `country_code` | -| buckets | The number of buckets to create while clustering | Required if `clustered_by` is specified | `8` | -| tblproperties | [Tblproperties](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-tblproperties.html) to be set on the created table | Optional | `{'this.is.my.key': 12}` | - - - - + | Option | Description | Required? | Model Support | Example | @@ -621,7 +608,7 @@ snapshots: - + ## Materialized views and streaming tables Starting with version 1.6.0, the dbt-databricks adapter supports [materialized views](https://docs.databricks.com/en/sql/user/materialized-views.html) and [streaming tables](https://docs.databricks.com/en/sql/load-data-streaming-table.html), as alternatives to incremental tables that are powered by [Delta Live Tables](https://docs.databricks.com/en/delta-live-tables/index.html). diff --git a/website/docs/reference/resource-configs/group.md b/website/docs/reference/resource-configs/group.md index 5fc5382760b..717d7de89f5 100644 --- a/website/docs/reference/resource-configs/group.md +++ b/website/docs/reference/resource-configs/group.md @@ -18,8 +18,6 @@ id: "group" }> - - ```yml @@ -60,14 +58,10 @@ select ... - - - - ```yml @@ -88,15 +82,10 @@ seeds: - - - - - ```yml @@ -123,15 +112,10 @@ select ... - - - - - ```yml @@ -184,14 +168,10 @@ select ... - - - - ```yml @@ -204,15 +184,11 @@ analyses: - - - - ```yaml @@ -237,8 +213,6 @@ metrics: - - diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index b7b88c525dd..07cfc938f1c 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -8,22 +8,10 @@ id: "postgres-configs" In dbt-postgres, the following incremental materialization strategies are supported: - - -- `append` (default when `unique_key` is not defined) -- `delete+insert` (default when `unique_key` is defined) - - - - - - `append` (default when `unique_key` is not defined) - `merge` - `delete+insert` (default when `unique_key` is defined) - - - ## Performance optimizations ### Unlogged @@ -104,8 +92,6 @@ models: - - ## Materialized views The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) @@ -200,7 +186,7 @@ This happens via a `DROP/CREATE` of the indexes, which can be thought of as an ` Learn more about these parameters in Postgres's [docs](https://www.postgresql.org/docs/current/sql-creatematerializedview.html). - + ### Limitations @@ -216,5 +202,3 @@ If the user changes the model's config to `materialized="materialized_view"`, th The solution is to execute `DROP TABLE my_model` on the data warehouse before trying the model again. - - diff --git a/website/docs/reference/resource-configs/redshift-configs.md b/website/docs/reference/resource-configs/redshift-configs.md index 78b288083fa..dcd87118d13 100644 --- a/website/docs/reference/resource-configs/redshift-configs.md +++ b/website/docs/reference/resource-configs/redshift-configs.md @@ -14,21 +14,10 @@ To-do: In dbt-redshift, the following incremental materialization strategies are supported: - - -- `append` (default when `unique_key` is not defined) -- `delete+insert` (default when `unique_key` is defined) - - - - - - `append` (default when `unique_key` is not defined) - `merge` - `delete+insert` (default when `unique_key` is defined) - - All of these strategies are inherited from dbt-postgres. ## Performance optimizations @@ -107,8 +96,6 @@ models: - - ## Materialized views The Redshift adapter supports [materialized views](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) @@ -241,7 +228,7 @@ As with most data platforms, there are limitations associated with materialized Find more information about materialized view limitations in Redshift's [docs](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-limitations). - + #### Changing materialization from "materialized_view" to "table" or "view" @@ -256,8 +243,6 @@ The workaround is to execute `DROP MATERIALIZED VIEW my_mv CASCADE` on the data - - ## Unit test limitations diff --git a/website/docs/reference/resource-configs/singlestore-configs.md b/website/docs/reference/resource-configs/singlestore-configs.md index 2317c05c0e2..c1a1c6869f1 100644 --- a/website/docs/reference/resource-configs/singlestore-configs.md +++ b/website/docs/reference/resource-configs/singlestore-configs.md @@ -107,8 +107,6 @@ select ... - - ## Model contracts Starting from 1.5, the `dbt-singlestore` adapter supports model contracts. @@ -217,5 +215,3 @@ It's important to note that certain data type mappings might show up differently Just keep these points in mind when setting up and using your `dbt-singlestore` adapter, and you'll avoid common pitfalls! - - diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 8123cb6b9eb..a59bc8dee00 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -338,8 +338,6 @@ In the configuration format for the model SQL file: - - ## Dynamic tables The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about). @@ -578,7 +576,7 @@ For dbt limitations, these dbt features are not supported: - [Model contracts](/docs/collaborate/govern/model-contracts) - [Copy grants configuration](/reference/resource-configs/snowflake-configs#copying-grants) - + #### Changing materialization to and from "dynamic_table" @@ -605,9 +603,6 @@ The workaround is to execute `DROP TABLE my_model` on the data warehouse before - - - ## Source freshness known limitation Snowflake calculates source freshness using information from the `LAST_ALTERED` column, meaning it relies on a field updated whenever any object undergoes modification, not only data updates. No action must be taken, but analytics teams should note this caveat. diff --git a/website/docs/reference/resource-configs/target_schema.md b/website/docs/reference/resource-configs/target_schema.md index 8af0daff42d..a130ce23f7f 100644 --- a/website/docs/reference/resource-configs/target_schema.md +++ b/website/docs/reference/resource-configs/target_schema.md @@ -40,9 +40,6 @@ On **BigQuery**, this is analogous to a `dataset`. ## Default This is a **required** parameter, no default is provided. -## FAQs - - ## Examples ### Build all snapshots in a schema named `snapshots` diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js index 1ee1a308546..9eb040e54fb 100644 --- a/website/docusaurus.config.js +++ b/website/docusaurus.config.js @@ -81,7 +81,7 @@ var siteSettings = { "https://www.getdbt.com/resources/webinars/dbt-cloud-demos-with-experts/?utm_medium=internal&utm_source=docs&utm_campaign=q2-2025_biweekly-demos_aw&utm_content=biweekly-demos____&utm_term=all_all__", // Set community spotlight member on homepage // This is the ID for a specific file under docs/community/spotlight - communitySpotlightMember: "tyler-rouze", + communitySpotlightMember: "meagan-palmer", prism: { theme: (() => { var theme = require("prism-react-renderer/themes/nightOwl"); diff --git a/website/sidebars.js b/website/sidebars.js index c5dea22053d..2890135fe9f 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -79,6 +79,7 @@ const sidebarSettings = { items: [ "docs/cloud/manage-access/about-user-access", "docs/cloud/manage-access/invite-users", + "docs/cloud/manage-access/mfa", { type: "category", label: "User permissions and licenses", diff --git a/website/snippets/_sl-deprecation-notice.md b/website/snippets/_sl-deprecation-notice.md deleted file mode 100644 index 2c42dd199c7..00000000000 --- a/website/snippets/_sl-deprecation-notice.md +++ /dev/null @@ -1,5 +0,0 @@ -:::info Deprecation of dbt Metrics and the legacy dbt Semantic Layer -dbt Labs has deprecated dbt Metrics and the legacy dbt Semantic Layer, both supported on dbt version 1.5 or lower. These changes went into effect on December 15th, 2023. - -To migrate and access [MetricFlow](/docs/build/build-metrics-intro) or the re-released dbt Semantic Layer, use the [dbt Semantic Layer migration guide](/guides/sl-migration) and [upgrade your version](/docs/dbt-versions/upgrade-dbt-version-in-cloud) in dbt Cloud. -::: diff --git a/website/snippets/cloud-feature-parity.md b/website/snippets/cloud-feature-parity.md index 6c6f224f215..b6c49aa0faa 100644 --- a/website/snippets/cloud-feature-parity.md +++ b/website/snippets/cloud-feature-parity.md @@ -1,17 +1,17 @@ The following table outlines which dbt Cloud features are supported on the different SaaS options available today. For more information about feature availability, please [contact us](https://www.getdbt.com/contact/). -| Feature | Multi-tenant | AWS single tenant | Azure single tenant | -|-------------------------------|--------------|-----------------------|----------------------| -| Audit logs | ✅ | ✅ | ✅ | -| Continuous integration jobs | ✅ | ✅ | ✅ | -| dbt Cloud CLI | ✅ | ✅ | ✅ | -| dbt Cloud IDE | ✅ | ✅ | ✅ | -| dbt Explorer | ✅ | ✅ | ✅ | -| dbt Mesh | ✅ | ✅ | ✅ | -| dbt Semantic Layer | ✅ | ✅ (Upon request) | ❌ | -| Discovery API | ✅ | ✅ | ✅ | -| IP restrictions | ✅ | ✅ | ✅ | -| Job scheduler | ✅ | ✅ | ✅ | -| PrivateLink egress | ✅ (AWS only)| ✅ | ✅ | -| PrivateLink ingress | ❌ | ✅ | ✅ | -| Webhooks (Outbound) | ✅ | ✅ | ❌ | +| Feature | AWS Multi-tenant | AWS single tenant |Azure multi-tenant ([Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud)) | Azure single tenant | +|-------------------------------|------------------|-----------------------|---------------------|------------------- -| +| Audit logs | ✅ | ✅ | ✅ | ✅ | +| Continuous integration jobs | ✅ | ✅ | ✅ | ✅ | +| dbt Cloud CLI | ✅ | ✅ | ✅ | ✅ | +| dbt Cloud IDE | ✅ | ✅ | ✅ | ✅ | +| dbt Explorer | ✅ | ✅ | ✅ | ✅ | +| dbt Mesh | ✅ | ✅ | ✅ | ✅ | +| dbt Semantic Layer | ✅ | ✅ (Upon request) | ✅ | ❌ | +| Discovery API | ✅ | ✅ | ✅ | ✅ | +| IP restrictions | ✅ | ✅ | ✅ | ✅ | +| Job scheduler | ✅ | ✅ | ✅ | ✅ | +| PrivateLink egress | ✅ (AWS only) | ✅ | ✅ | ✅ | +| PrivateLink ingress | ❌ | ✅ | ❌ | ✅ | +| Webhooks (Outbound) | ✅ | ✅ | ✅ | ❌ | diff --git a/website/snippets/core-versions-table.md b/website/snippets/core-versions-table.md index 13a77f6dd30..b9f79f7d76a 100644 --- a/website/snippets/core-versions-table.md +++ b/website/snippets/core-versions-table.md @@ -4,7 +4,7 @@ |:-------------------------------------------------------------:|:---------------:|:-------------------------------------:| | [**v1.8**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8) | May 9 2024 | Active — May 8, 2025 | | [**v1.7**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.7) | Nov 2, 2023 | Critical — Nov 1, 2024 | -| [**v1.6**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.6) | Jul 31, 2023 | Critical — Jul 30, 2024 | +| [**v1.6**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.6) | Jul 31, 2023 | End of Life* ⚠️ | | [**v1.5**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.5) | Apr 27, 2023 | End of Life* ⚠️ | | [**v1.4**](/docs/dbt-versions/core-upgrade/older%20versions/upgrading-to-v1.4) | Jan 25, 2023 | End of Life* ⚠️ | | [**v1.3**](/docs/dbt-versions/core-upgrade/older%20versions/upgrading-to-v1.3) | Oct 12, 2022 | End of Life* ⚠️ | diff --git a/website/src/components/communitySpotlightList/index.js b/website/src/components/communitySpotlightList/index.js index c52fe7891c7..8355fd0958b 100644 --- a/website/src/components/communitySpotlightList/index.js +++ b/website/src/components/communitySpotlightList/index.js @@ -11,7 +11,7 @@ const communityDescription = "The dbt Community is where analytics engineering l // This date determines where the 'Previously on the Spotlight" text will show. // Any spotlight members with a 'dateCreated' field before this date // will be under the 'Previously..' header. -const currentSpotlightDate = new Date('2024-05-01') +const currentSpotlightDate = new Date('2024-07-26') function CommunitySpotlightList({ spotlightData }) { const { siteConfig } = useDocusaurusContext() diff --git a/website/static/img/community/spotlight/Meagan-Palmer.png b/website/static/img/community/spotlight/Meagan-Palmer.png new file mode 100644 index 00000000000..b24bf8db484 Binary files /dev/null and b/website/static/img/community/spotlight/Meagan-Palmer.png differ diff --git a/website/static/img/community/spotlight/Mikko-Sulonen.png b/website/static/img/community/spotlight/Mikko-Sulonen.png new file mode 100644 index 00000000000..faba53b639b Binary files /dev/null and b/website/static/img/community/spotlight/Mikko-Sulonen.png differ diff --git a/website/static/img/community/spotlight/Radovan-Bacovic.png b/website/static/img/community/spotlight/Radovan-Bacovic.png new file mode 100644 index 00000000000..f4e98bede9d Binary files /dev/null and b/website/static/img/community/spotlight/Radovan-Bacovic.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png new file mode 100644 index 00000000000..8e02ba262fe Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png new file mode 100644 index 00000000000..c4754e4bf60 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png new file mode 100644 index 00000000000..8e02ba262fe Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png new file mode 100644 index 00000000000..c4754e4bf60 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png differ