diff --git a/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md b/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md index 5098e9d29e3..f7e5786bc70 100644 --- a/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md +++ b/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md @@ -18,7 +18,7 @@ It's been a few years since dbt-core turned 1.0! Since then, we've committed to In 2024, we're taking this promise further by: - Stabilizing interfaces for everyone — adapter maintainers, metadata consumers, and (of course) people writing dbt code everywhere — as discussed in [our November 2023 roadmap update](https://github.com/dbt-labs/dbt-core/blob/main/docs/roadmap/2023-11-dbt-tng.md). -- Introducing **Keep on latest version** in dbt Cloud. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud). +- Introducing **Versionless** in dbt Cloud. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud). We're leaving the rest of this post as is, so we can all remember how it used to be. Enjoy a stroll down memory lane. diff --git a/website/blog/2024-04-22-extended-attributes.md b/website/blog/2024-04-22-extended-attributes.md index d00eb57569d..18d4ff0b64c 100644 --- a/website/blog/2024-04-22-extended-attributes.md +++ b/website/blog/2024-04-22-extended-attributes.md @@ -80,7 +80,7 @@ All you need to do is configure an environment as staging and enable the **Defer ## Upgrading on a curve -Lastly, let’s consider a more specialized use case. Imagine we have a "tiger team" (consisting of a lone analytics engineer named Dave) tasked with upgrading from dbt version 1.6 to the new **Keep on latest version** setting, to take advantage of added stability and feature access. We want to keep the rest of the data team being productive in dbt 1.6 for the time being, while enabling Dave to upgrade and do his work in the new versionless mode. +Lastly, let’s consider a more specialized use case. Imagine we have a "tiger team" (consisting of a lone analytics engineer named Dave) tasked with upgrading from dbt version 1.6 to the new **Versionless** setting, to take advantage of added stability and feature access. We want to keep the rest of the data team being productive in dbt 1.6 for the time being, while enabling Dave to upgrade and do his work in the new versionless mode. ### Development environment diff --git a/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md b/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md index 0953e9b3c27..078dab198fa 100644 --- a/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md +++ b/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md @@ -1,5 +1,5 @@ --- -title: "How we're making sure you can confidently \"Keep on latest version\" in dbt Cloud" +title: "How we're making sure you can confidently go \"Versionless\" in dbt Cloud" description: "Over the past 6 months, we've laid a stable foundation for continuously improving dbt." slug: latest-dbt-stability @@ -16,19 +16,19 @@ As long as dbt Cloud has existed, it has required users to select a version of d However, this came at a cost. While bumping a project's dbt version *appeared* as simple as selecting from a dropdown, there was real effort required to test the compatibility of the new version against existing projects, package dependencies, and adapters. On the other hand, putting this off meant foregoing access to new features and bug fixes in dbt. -But no more. Today, we're ready to announce the general availability of a new option in dbt Cloud: [**"Keep on latest version."**](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) +But no more. Today, we're ready to announce the general availability of a new option in dbt Cloud: [**"Versionless."**](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) For customers, this means less maintenance overhead, faster access to bug fixes and features, and more time to focus on what matters most: building trusted data products. This will be our stable foundation for improvement and innovation in dbt Cloud. -But we wanted to go a step beyond just making this option available to you. In this blog post, we aim to shed a little light on the extensive work we've done to ensure that using "Keep on latest version" is a stable, reliable experience for the thousands of customers who rely daily on dbt Cloud. +But we wanted to go a step beyond just making this option available to you. In this blog post, we aim to shed a little light on the extensive work we've done to ensure that using "Versionless" is a stable, reliable experience for the thousands of customers who rely daily on dbt Cloud. ## How we safely deploy dbt upgrades to Cloud We've put in place a rigorous, best-in-class suite of tests and control mechanisms to ensure that all changes to dbt under the hood are fully vetted before they're deployed to customers of dbt Cloud. -This pipeline has in fact been in place since January! It's how we've already been shipping continuous changes to the hundreds of customers who've selected "Keep on latest version" while it's been in Beta and Preview. In that time, this process has enabled us to prevent multiple regressions before they were rolled out to any customers. +This pipeline has in fact been in place since January! It's how we've already been shipping continuous changes to the hundreds of customers who've selected "Versionless" while it's been in Beta and Preview. In that time, this process has enabled us to prevent multiple regressions before they were rolled out to any customers. We're very confident in the robustness of this process**. We also know that we'll need to continue building trust with time.** We're sharing details about this work in the spirit of transparency and to build that trust. @@ -82,9 +82,9 @@ All incidents are retrospected to make sure we not only identify and fix the roo ::: -The outcome of this process is that, when you select "Keep on latest version" in dbt Cloud, the time between an improvement being made to dbt Core and you *safely* getting access to it in your projects is a matter of days — rather than months of waiting for the next dbt Core release, on top of any additional time it may have taken to actually carry out the upgrade. +The outcome of this process is that, when you select "Versionless" in dbt Cloud, the time between an improvement being made to dbt Core and you *safely* getting access to it in your projects is a matter of days — rather than months of waiting for the next dbt Core release, on top of any additional time it may have taken to actually carry out the upgrade. -We’re pleased to say that since the beta launch of “Keep on latest version” in dbt Cloud in March, **we have not had any functional regressions reach customers**, while we’ve also been shipping multiple improvements to dbt functionality every day. This is a foundation that we aim to build on for the foreseeable future. +We’re pleased to say that since the beta launch of “Versionless” in dbt Cloud in March, **we have not had any functional regressions reach customers**, while we’ve also been shipping multiple improvements to dbt functionality every day. This is a foundation that we aim to build on for the foreseeable future. ## Stability as a feature @@ -98,7 +98,7 @@ The adapter interface — i.e. how dbt Core actually connects to a third-party d To solve that, we've released a new set of interfaces that are entirely independent of the `dbt-core` library: [`dbt-adapters==1.0.0`](https://github.com/dbt-labs/dbt-adapters). From now on, any changes to `dbt-adapters` will be backward and forward-compatible. This also decouples adapter maintenance from the regular release cadence of dbt Core — meaning maintainers get full control over when they ship implementations of new adapter-powered features. -Note that adapters running in dbt Cloud **must** be [migrated to the new decoupled architecture](https://github.com/dbt-labs/dbt-adapters/discussions/87) as a baseline in order to support the new "Keep on latest version". +Note that adapters running in dbt Cloud **must** be [migrated to the new decoupled architecture](https://github.com/dbt-labs/dbt-adapters/discussions/87) as a baseline in order to support the new "Versionless" option. ### Managing behavior changes: stability as a feature @@ -118,7 +118,7 @@ We’ve now [formalized our development best practices](https://github.com/dbt-l In conclusion, we’re putting a lot of new muscle behind our commitments to dbt Cloud customers, the dbt Community, and the broader ecosystem: -- **Continuous updates**: "Keep on latest version" in dbt Cloud simplifies the update process, ensuring you always have the latest features and bug fixes without the maintenance overhead. +- **Continuous updates**: "Versionless" dbt Cloud simplifies the update process, ensuring you always have the latest features and bug fixes without the maintenance overhead. - **A rigorous new testing and deployment process**: Our new testing pipeline ensures that every update is carefully vetted against documented interfaces, Cloud-supported adapters, and popular packages before it reaches you. This process minimizes the risk of regressions — and has now been successful at entirely preventing them for hundreds of customers over multiple months. - **A commitment to stability**: We’ve reworked our approaches to adapter interfaces, behaviour change management, and metadata artifacts to give you more stability and control. diff --git a/website/blog/2024-06-12-putting-your-dag-on-the-internet.md b/website/blog/2024-06-12-putting-your-dag-on-the-internet.md index 8d0bc79e35d..535cfc34d6e 100644 --- a/website/blog/2024-06-12-putting-your-dag-on-the-internet.md +++ b/website/blog/2024-06-12-putting-your-dag-on-the-internet.md @@ -114,6 +114,6 @@ Traditionally dbt is the T in ELT (dbt overview [here](https://docs.getdbt.com/t In order to get this functionality shipped quickly, EQT opened a pull request, Snowflake helped with some problems we had with CI and a member of dbt Labs helped write the tests and merge the code in! -dbt now features this functionality in dbt 1.8+ or on “Keep on latest version” option of dbt Cloud (dbt overview [here](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)). +dbt now features this functionality in dbt 1.8+ or the “Versionless” option of dbt Cloud (dbt overview [here](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless)). dbt Labs staff and community members would love to chat more about it in the [#db-snowflake](https://getdbt.slack.com/archives/CJN7XRF1B) slack channel. diff --git a/website/blog/2024-07-24-cityblock-merge-jobs.md b/website/blog/2024-07-24-cityblock-merge-jobs.md deleted file mode 100644 index 228ab17615f..00000000000 --- a/website/blog/2024-07-24-cityblock-merge-jobs.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: "Practitioner Q&A: How Cityblock Health adopted dbt Cloud's Merge Jobs to deploy new code faster" -description: "Cityblock Health swapped their GitHub Actions for native dbt Cloud functionality to streamline their project deployments and avoid redundant resouce consumption." -slug: cityblock-merge-jobs -authors: [nathaniel_burren, joel_labes] -tags: [dbt Cloud] -hide_table_of_contents: false -date: 2024-07-30 -is_featured: true ---- - -dbt Labs recently added support for a new type of job in dbt Cloud, called a **[Merge Job](/docs/deploy/merge-jobs)**. Just like a Continuous Integration job is triggered based on a new Pull Request being opened in your git repository, the Merge Job triggers once code is merged into your environment’s configured branch (e.g. `main` for the Production environment, or `uat` for the Staging environment). - -Triggering a run when code is merged enables your team to practice Continuous Deployment, getting changed models deployed as quickly as possible and reducing resource consumption from unnecessary overbuilding. Alternatively, just kick off a `dbt compile` to update your project’s state for Slim CI comparisons. Check out the [Trigger on Merge docs](/docs/deploy/merge-jobs) for full details on enabling this in your project. - -The Cityblock Health team were one of the first companies to use Merge Jobs during the beta period, and I (Joel) caught up with Analytics Engineer Nathaniel Burren to hear how they did it and what benefits they’ve seen. - - - -## How long have you been using dbt at Cityblock? - -Cityblock chose dbt in 2019 as the tool to make all of our SQL based analysis version controlled, composable, and reusable. However, over the years our monolithic project turned into a jungle of over 2,200 models. This made us rethink our approach and move to dbt Cloud while also adapting a [multi-project methodology](/best-practices/how-we-mesh/mesh-1-intro). We presented [our migration story at Coalesce last year](https://www.youtube.com/watch?v=oO7whNtd9Jg) and go into more details there. - -Our hope is to create core assets maintained by our Analytics Engineers that our downstream teams can use in their own projects. This also allows us to retain control of these core assets as our single sources of truth while the downstream projects can create their own assets based on their subject matter expertise (that in turn other projects can use, instead of having a mess of duplicate models). - -## What made you excited enough about Merge Jobs to sign up for the beta? - -Our goal when adopting Merge Jobs was to get model changes in our Staging and Production environments deployed as fast as possible. We also like to have fresh Explorer data as we have many eyes from downstream users looking at the core models in our data platform. Keeping the environments up to date with our git repo means that when we fix something the resolution is immediately deployed, not languishing waiting for our overnight build. - -## What were you doing to achieve these goals before? - -We used to use a GitHub Action to trigger a production run on a merge to main. However most of our Analytics Engineerss don’t understand GHAs and we would have to get our DevOps friends to help us whenever configuration changes were needed. In turn that would cause us to have to wait on them or an AE that knows GHAs to help change/maintain the code. - -## And what are you doing now? How did you configure your Merge Job? - -Because we have so many dbt projects split across different domains, one of my colleagues built a Terraform module which builds new dbt Cloud projects, including environments, jobs, and permissions. We added the Merge Job to that definition and deployed it everywhere in one go. - -The job itself only has one step, `dbt build --select state:modified+`. It defers to itself, so every time a commit is merged to main the only things that get built are models that changed compared to that last run. - -## How has switching to the native functionality improved things for you? - -The biggest place this has helped us is no longer having to babysit deploy jobs. With the fast pace we’re working at right now, it really helps us keep working on other things and not worry about manually triggering jobs. - -## What advice would you give to other teams who are implementing Merge Jobs? - -Make sure you’ve got your [branch protection rules in GitHub](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule#creating-a-branch-protection-rule) locked down. We noticed that some downstream project users were merging PRs even though their [dbt-project-evaluator checks](/blog/align-with-dbt-project-evaluator) were failing. Since failed jobs aren’t eligible for deferral, this caused some weird behavior for us until we sorted it out. diff --git a/website/dbt-versions.js b/website/dbt-versions.js index 7709fea304c..5ad3a3048c5 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -15,10 +15,6 @@ exports.versions = [ version: "1.6", EOLDate: "2024-07-31", }, - { - version: "1.5", - EOLDate: "2024-04-27", - }, ] exports.versionedPages = [ diff --git a/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md b/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md index e57ed5c966f..b6fadc2d7a6 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md +++ b/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md @@ -20,8 +20,12 @@ Is dbt Mesh a good fit in this scenario? Absolutely! There is no other way to sh ### Adoption challenges -- Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. dbt Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that dbt Mesh will help your teams deliver concrete value quickly. -- Bi-directional project dependencies -- currently, projects in dbt Mesh are treated like dbt resources in that they cannot depend on each other. In reality, domain teams likely need the ability to have “chatty” APIs; otherwise, they need to split projects beyond the 1:1 mapping with team boundaries. While this constraint exists today, we're working to remove this point of friction. More information about this will be provided in the near future! +- Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. dbt Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. + + Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that dbt Mesh will help your teams deliver concrete value quickly. +- Bi-directional project dependencies -- currently, projects in dbt Mesh are treated like dbt resources in that they cannot depend on each other. However, many teams may want to be able to share data assets back and forth between teams. + + We've added support for [enabling bidirectional dependencies](/best-practices/how-we-mesh/mesh-3-structures#cycle-detection) across projects. If this sounds like your organization, dbt Mesh is the architecture you should pursue. ✅ diff --git a/website/docs/best-practices/how-we-mesh/mesh-3-structures.md b/website/docs/best-practices/how-we-mesh/mesh-3-structures.md index 91090521e9f..c75c566610b 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-3-structures.md +++ b/website/docs/best-practices/how-we-mesh/mesh-3-structures.md @@ -18,14 +18,6 @@ At a high level, you’ll need to decide: - Where to draw the lines between your dbt Projects -- i.e. how do you determine where to split your DAG and which models go in which project? - How to manage your code -- do you want multiple dbt Projects living in the same repository (mono-repo) or do you want to have multiple repos with one repo per project? -### Cycle detection - -Like resource dependencies, project dependencies are acyclic, meaning they only move in one direction. This prevents `ref` cycles (or loops), which lead to issues with your data workflows. For example, if project B depends on project A, a new model in project A could not import and use a public model from project B. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref) for more information. - -:::note -This requirement is likely to change in the future, so stay tuned for updates! -::: - ## Define your project interfaces by splitting your DAG The first (and perhaps most difficult!) decision when migrating to a multi-project architecture is deciding where to draw the line in your DAG to define the interfaces between your projects. Let's explore some language for discussing the design of these patterns. @@ -74,6 +66,13 @@ Since the launch of dbt Mesh, the most common pattern we've seen is one where pr Users may need to contribute models across multiple projects and this is fine. There will be some friction doing this, versus a single repo, but this is _useful_ friction, especially if upstreaming a change from a “spoke” to a “hub.” This should be treated like making an API change, one that the other team will be living with for some time to come. You should be concerned if your teammates find they need to make a coordinated change across multiple projects very frequently (every week), or as a key prerequisite for ~20%+ of their work. +### Cycle detection + +import CycleDetection from '/snippets/_mesh-cycle-detection.md'; + + + + ### Tips and tricks The [implementation](/best-practices/how-we-mesh/mesh-4-implementation) page provides more in-depth examples of how to split a monolithic project into multiple projects. Here are some tips to get you started when considering the splitting methods listed above on your own projects: diff --git a/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md b/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md index 980228f4ed4..f52cff78188 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md +++ b/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md @@ -78,7 +78,10 @@ dbt Mesh also enhances the interoperability and reusability of data across diffe -Like resource dependencies, project dependencies are acyclic, meaning they only move in one direction. This prevents `ref` cycles (or loops). For example, if project B depends on project A, a new model in project A could not import and use a public model from project B. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies#how-to-use-ref) for more information. +import CycleDetection from '/snippets/_mesh-cycle-detection.md'; + + + diff --git a/website/docs/community/resources/oss-expectations.md b/website/docs/community/resources/oss-expectations.md index ba6c798e843..e6e5d959c96 100644 --- a/website/docs/community/resources/oss-expectations.md +++ b/website/docs/community/resources/oss-expectations.md @@ -107,3 +107,7 @@ PRs are your surest way to make the change you want to see in dbt / packages / d - **Tests!** When you open a PR, some tests and code checks will run. (For security reasons, some may need to be approved by a maintainer.) We will not merge any PRs with failing tests. If you’re not sure why a test is failing, please say so, and we’ll do our best to get to the bottom of it together. - **Contributor License Agreement** (CLA): This ensures that we can merge your code, without worrying about unexpected implications for the copyright or license of open source dbt software. For more details, read: ["Contributor License Agreements"](../resources/contributor-license-agreements.md) - **Changelog:** In projects that include a number of changes in each release, we need a reliable way to signal what's been included. The mechanism for this will vary by repository, so keep an eye out for notes about how to update the changelog. + +### Inclusion in release versions + +Both bug fixes and backwards-compatible new features will be included in the [next minor release](/docs/dbt-versions/core#how-dbt-core-uses-semantic-versioning). Fixes for regressions and net-new bugs that were present in the minor version's original release will be backported to versions with [active support](/docs/dbt-versions/core). Other bug fixes may be backported when we have high confidence that they're narrowly scoped and won't cause unintended side effects. diff --git a/website/docs/community/spotlight/meagan-palmer.md b/website/docs/community/spotlight/meagan-palmer.md new file mode 100644 index 00000000000..ff45a3d6b7d --- /dev/null +++ b/website/docs/community/spotlight/meagan-palmer.md @@ -0,0 +1,41 @@ +--- +id: meagan-palmer +title: Meagan Palmer +description: | + I first started using dbt in 2016 or 2017 (I can't remember exactly). Since then, I have moved into data and analytics consulting and have dipped in and out of the dbt Community. + Late last year, I started leading dbt Cloud training courses and spending more time in the dbt Slack. + In consulting, I get to use a range of stacks. I've used dbt with Redshift, Snowflake, and Databricks in production settings with a range of loaders & reporting tools, and I've been enjoying using DuckDB for some home experimentation. + To share some of the experiences, I regularly post to LinkedIn and have recently started Analytics Engineering Today, a twice monthly newsletter about dbt in practice. +image: /img/community/spotlight/Meagan-Palmer.png +pronouns: she/her +location: Sydney, Australia +jobTitle: Principal Consultant +companyName: Altis Consulting +socialLinks: + - name: LinkedIn + link: https://www.linkedin.com/in/meaganpalmer/ +dateCreated: 2024-07-29 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I was fortunate that Jon Bradley at Nearmap had the vision to engage the then Fishtown Analytics team (as the dbt Labs team was formerly called) as consultants and begin using dbt in our stack. I can't thank him enough. It was a turning point for my career, where I could combine my interests and experiences in delivering business value, data, product management, and software engineering practices. + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +Being in Australia, I often see replies from Jeremy Yeo to people in the dbt Slack. His clarity of communication is impressive. + +For growth, I'm hoping that others can benefit from the wide range of experience I have. My newsletter, Analytics Engineering Today on LinkedIn aims to upskill the dbt Community and shed some light on some useful features that might not be well known. + +I'll be at Coalesce and am doing some webinars/events later in the year. Come say hi, I love talking dbt and analytics engineering with people. + +## What have you learned from community members? What do you hope others can learn from you? + +The community members are amazing. It's great to be among a group of people that want to learn and improve. +I've learned a lot - both from other members helping with my queries and in reading how other businesses have implemented dbt, including their stories on the organizational & technical issues they face. + +I hope I can help instill a sense that simple, clean solutions are possible and preferable. I want to highlight that it is important to focus on what is the actual problem you are trying to solve and the fact that it's worth asking for help when you're starting to get stuck. + +I'm keen to help more women get the courage to work & lead in STEM. There has been a lot of progress made over the course of my career which is great to see. Australian/NZ women, please connect with me, happy to chat. diff --git a/website/docs/community/spotlight/mikko-sulonen.md b/website/docs/community/spotlight/mikko-sulonen.md new file mode 100644 index 00000000000..689b53db97e --- /dev/null +++ b/website/docs/community/spotlight/mikko-sulonen.md @@ -0,0 +1,33 @@ +--- +id: mikko-sulonen +title: Mikko Sulonen +description: | + I've been working with data since 2016. I first started with the on-prem SQL Server S-stack of SSIS, SSAS, SSRS. I did some QlikView and Qlik Sense, and some Power BI. Nowadays, I work mostly with Snowflake, Databricks, Azure, and dbt, of course. While tools and languages have come and gone, SQL has stayed. I've been a consultant for all of my professional life. +image: /img/community/spotlight/Mikko-Sulonen.png +pronouns: he/him +location: Tampere, Finland +jobTitle: Data Architect, Partner +companyName: Recordly Oy +socialLinks: + - name: LinkedIn + link: https://www.linkedin.com/in/mikkosulonen/ +dateCreated: 2024-07-28 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I started using dbt around 2019-2020, I think. Snapshots (no longer "archives") were a new thing along with the RPC server! I asked around my then-company: pretty much nobody had used dbt, though some commented that it did look promising. That left me looking elsewhere for experiences and best practices around the tool, and I found different blog writers and eventually the [dbt Slack](https://www.getdbt.com/community/join-the-community). I quickly noticed I could learn much more from the experiences of others than by trying everything myself. After just lurking for a while, I started to answer people's questions and give my own thoughts. This was completely new to me: voicing my input and opinions to people I had never met or who were not my colleagues. + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +There are quite many. I started to write some names here, but felt the list was getting a bit long and I'd still forget somebody! What the community leaders have in common is that they are approachable, knowledgeable, and passionate. They want to help others, they want to drive the community forward, and they are down to earth. I've had the pleasure of meeting many of them in person at the past two [Coalesces](https://coalesce.getdbt.com/), and I hope to meet many more! + +Growing my own leadership in the community... That's an interesting question: I hadn't really identified myself as leader in the community before. Maybe I should come out of the Slack channels and join and/or organize some [dbt Meetups](https://www.meetup.com/pro/dbt/)? I always try to answer even the simplest questions, even if they've been answered a hundred times already. Every day, new people are introduced to dbt, and they are facing issues for the first time. Every one of us was new at one point! + +## What have you learned from community members? What do you hope others can learn from you? + +I've learnt a surprising amount about the different teams and ways of working with and around data. I've learnt that it is highly probable that somebody somewhere has already had, and likely solved, the problem you are having. All that is needed to connect the dots is for people to speak and listen. + +When asking and answering questions, I try to hone in on what they're really trying to get at. I ask, why is it that you want to do something in that particular way? I like to say, don't love the solutions, love the problems! diff --git a/website/docs/community/spotlight/radovan-bacovic.md b/website/docs/community/spotlight/radovan-bacovic.md new file mode 100644 index 00000000000..186edccac6d --- /dev/null +++ b/website/docs/community/spotlight/radovan-bacovic.md @@ -0,0 +1,42 @@ +--- +id: radovan-bacovic +title: Radovan Bacovic +description: | + My professional journey and friendship with data started 20 years ago. + I've experienced many tools and modalities: from good old RDMS systems and various programming languages like Java and C# in the early days of my career, through private clouds, to MPP databases and multitier architecture. I also saw the emergence of cloud technology, and experienced the changes that came with it, up to the contemporary approach to data, which includes AI tools and practices. I am still excited to get new knowledge and solve problems in the data world. I always enjoy using SQL and Python as my primary "weapons" of choice together with other building blocks for successful data products like Snowflake, dbt, Tableau, Fivetran, Stich, Monte Carlo, DuckDB and more. + I consider myself as an experienced data engineer and a wannabe "best bad Conference speaker." +image: /img/community/spotlight/Radovan-Bacovic.png +pronouns: he/him +location: Novi Sad, Serbia +jobTitle: Staff Data Engineer +companyName: GitLab +organization: Permifrost maintainer +socialLinks: + - name: X + link: https://x.com/svenmurtinson + - name: LinkedIn + link: https://www.linkedin.com/in/radovan-ba%C4%87ovi%C4%87-6498603/ + - name: Gitlab Profile + link: https://gitlab.com/rbacovic +dateCreated: 2024-07-28 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I have been in the dbt Community for almost 3 years now. The biggest impact that dbt has had on my professional journey is that it has given me a trustworthy partner for data transformation, and a single source of truth for all our data modification needs. As a public speaker who travels internationally, I recognized the interest of the data community in dbt around the world and, in response, organised several workshops and talks to help people use dbt. Let's just say that jumping into a great partnership with the dbt Community has been the greatest takeaway for me! + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +A leader from the dbt Commuity who I have found to be the most prominent is Sean McIntyre from dbt Labs, as I've had the privilege to collaborate with him many times. We recognized that we had a similar passion and energy; it looks like we are on the same journey. + +I wanted to be more involved in the dbt Community, spread the word, and share my journey through tutorials, conference talks, blogs and meetups. I think I am capable of addressing my influence in that direction. I am also interested in extending the dbt functionality and automating the deployment, testing and execution of dbt. In other words, I try to find good ways to leverage using DevSecOps for dbt to make our development faster and make dbt our trustworthy partner. + +## What have you learned from community members? What do you hope others can learn from you? + +Being part of the community is always a two-way street and always has a positive result for all of us. Great ideas and sharing vision and energy are the number one things. On the other side, I always find the quick and "best in class" answer to my questions from the community and try to return the favour and be helpful whenever possible. As I said, through talks and tutorials, I think I am the most beneficial community member in that role. + +## Anything else interesting you want to tell us? + +Nothing to add. Still passionate about discovering the tool and can't wait to see what the AI uprising will bring to us! diff --git a/website/docs/docs/build/custom-aliases.md b/website/docs/docs/build/custom-aliases.md index 9e9f91f968d..4f22de63e3f 100644 --- a/website/docs/docs/build/custom-aliases.md +++ b/website/docs/docs/build/custom-aliases.md @@ -6,18 +6,22 @@ id: "custom-aliases" ## Overview -When dbt runs a model, it will generally create a relation (either a `table` or a `view`) in the database. By default, dbt uses the filename of the model as the identifier for this relation in the database. This identifier can optionally be overridden using the [`alias`](/reference/resource-configs/alias) model configuration. +When dbt runs a model, it will generally create a relation (either a or a ) in the database, except in the case of an [ephemeral model](/docs/build/materializations), when it will create a for use in another model. By default, dbt uses the model's filename as the identifier for the relation or CTE it creates. This identifier can be overridden using the [`alias`](/reference/resource-configs/alias) model configuration. ### Why alias model names? The names of schemas and tables are effectively the "user interface" of your . Well-named schemas and tables can help provide clarity and direction for consumers of this data. In combination with [custom schemas](/docs/build/custom-schemas), model aliasing is a powerful mechanism for designing your warehouse. +The file naming scheme that you use to organize your models may also interfere with your data platform's requirements for identifiers. For example, you might wish to namespace your files using a period (`.`), but your data platform's SQL dialect may interpret periods to indicate a separation between schema names and table names in identifiers, or it may forbid periods from being used at all in CTE identifiers. In cases like these, model aliasing can allow you to retain flexibility in the way you name your model files without violating your data platform's identifier requirements. + ### Usage -The `alias` config can be used to change the name of a model's identifier in the database. The following shows examples of database identifiers for models both with, and without, a supplied `alias`. +The `alias` config can be used to change the name of a model's identifier in the database. The following table shows examples of database identifiers for models both with and without a supplied `alias`, and with different materializations. -| Model | Config | Database Identifier | -| ----- | ------ | ------------------- | -| ga_sessions.sql | <None> | "analytics"."ga_sessions" | -| ga_sessions.sql | {{ config(alias='sessions') }} | "analytics"."sessions" | +| Model | Config | Relation Type | Database Identifier | +| ----- | ------ | --------------| ------------------- | +| ga_sessions.sql | {{ config(materialization='view') }} | | "analytics"."ga_sessions" | +| ga_sessions.sql | {{ config(materialization='view', alias='sessions') }} | | "analytics"."sessions" | +| ga_sessions.sql | {{ config(materialization='ephemeral') }} | | "\__dbt\__cte\__ga_sessions" | +| ga_sessions.sql | {{ config(materialization='ephemeral', alias='sessions') }} | | "\__dbt\__cte\__sessions" | To configure an alias for a model, supply a value for the model's `alias` configuration parameter. For example: @@ -73,8 +77,6 @@ To override dbt's alias name generation, create a macro named `generate_alias_na The default implementation of `generate_alias_name` simply uses the supplied `alias` config (if present) as the model alias, otherwise falling back to the model name. This implementation looks like this: - - ```jinja2 @@ -100,8 +102,6 @@ The default implementation of `generate_alias_name` simply uses the supplied `al - - import WhitespaceControl from '/snippets/_whitespace-control.md'; diff --git a/website/docs/docs/build/custom-databases.md b/website/docs/docs/build/custom-databases.md index 2d430033b66..7a607534230 100644 --- a/website/docs/docs/build/custom-databases.md +++ b/website/docs/docs/build/custom-databases.md @@ -89,14 +89,10 @@ import WhitespaceControl from '/snippets/_whitespace-control.md'; - - ### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) - - ## Considerations ### BigQuery diff --git a/website/docs/docs/build/custom-schemas.md b/website/docs/docs/build/custom-schemas.md index 846f2d1c341..6dabf56943c 100644 --- a/website/docs/docs/build/custom-schemas.md +++ b/website/docs/docs/build/custom-schemas.md @@ -154,14 +154,10 @@ The following context methods _are_ available in the `generate_schema_name` macr Globally-scoped variables and variables defined on the command line with [--vars](/docs/build/project-variables) are accessible in the `generate_schema_name` context. - - ### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) - - ## A built-in alternative pattern for generating schema names A common customization is to ignore the target schema in production environments, and ignore the custom schema configurations in other environments (such as development and CI). diff --git a/website/docs/docs/build/data-tests.md b/website/docs/docs/build/data-tests.md index 38a2c74b646..158b8427136 100644 --- a/website/docs/docs/build/data-tests.md +++ b/website/docs/docs/build/data-tests.md @@ -19,7 +19,7 @@ keywords: :::important -In dbt v1.8, what was previously known as "tests" are now called "data tests" with the addition of [unit tests](/docs/build/unit-tests). The YAML key `tests:` is still supported as an alias for data tests but will be deprecated in the future in favor of `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. +From dbt v1.8, "tests" are now called "data tests" to disambiguate from [unit tests](/docs/build/unit-tests). The YAML key `tests:` is still supported as an alias for `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. ::: @@ -275,13 +275,11 @@ In dbt version 1.8, we updated the `tests` configuration to `data_tests`. For de - + -Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, it was necessary to update our naming conventions and syntax. - -As of v1.8, `tests:` is still supported in your YML configuration files as an alias but will be deprecated in the future in favor of `data_tests:`. +Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, the key was renamed from `tests:` to `data_tests:`. -As we progress towards this deprecation, we'll update the examples in our docs pages to reflect this new syntax, but we highly recommend you begin the migration process as soon as you upgrade to v1.8 to avoid interruptions or issues in the future. +dbt still supports `tests:` in your YML configuration files for backwards-compatibility purposes, and you might see it used throughout our documentation. However, you can't have a `tests` and a `data_tests` key associated with the same resource (e.g. a single model) at the same time. @@ -306,6 +304,8 @@ data_tests: +To suppress warnings about the rename, add `TestsConfigDeprecation` to the `silence` block of the `warn_error_options` flag in `dbt_project.yml`, [as described in the Warnings documentation](https://docs.getdbt.com/reference/global-configs/warnings). + ## FAQs diff --git a/website/docs/docs/build/environment-variables.md b/website/docs/docs/build/environment-variables.md index 664a7a6cf15..01601ce7eb8 100644 --- a/website/docs/docs/build/environment-variables.md +++ b/website/docs/docs/build/environment-variables.md @@ -9,7 +9,7 @@ Environment variables can be used to customize the behavior of a dbt project dep :::info Environment Variable Naming and Prefixing -Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET` or `DBT_ENV_CUSTOM_ENV_`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. +Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET` or `DBT_ENV_CUSTOM_ENV_`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. ::: @@ -86,7 +86,7 @@ There are some known issues with partial parsing of a project and changing envir ### Handling secrets -While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in the UI, you can prefix the key with `DBT_ENV_SECRET_``DBT_ENV_SECRET`. This functionality is supported from `dbt v1.0` and on. +While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in the UI, you can prefix the key with `DBT_ENV_SECRET`. This functionality is supported from `dbt v1.0` and on. @@ -97,8 +97,6 @@ While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud ha dbt Cloud has a number of pre-defined variables built in. Variables are set automatically and cannot be changed. - - **dbt Cloud IDE details** The following environment variable is set automatically for the dbt Cloud IDE: @@ -111,8 +109,6 @@ The following environment variable is set automatically for the dbt Cloud IDE: Use case — This is useful in cases where you want to dynamically use the Git branch name as a prefix for a [development schema](/docs/build/custom-schemas) ( `{{ env_var ('DBT_CLOUD_GIT_BRANCH') }}` ). - - **dbt Cloud context** The following environment variables are set automatically for deployment runs: diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 21cd656484a..df95504ceab 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -142,7 +142,7 @@ from {{ ref('app_data_events') }} -- this filter will only be applied on an incremental run -- (uses >= to include records arriving later on the same day as the last run of this model) - where date_day >= (select coalesce(max(event_time), '1900-01-01') from {{ this }}) + where date_day >= (select coalesce(max(date_day), '1900-01-01') from {{ this }}) {% endif %} diff --git a/website/docs/docs/build/incremental-strategy.md b/website/docs/docs/build/incremental-strategy.md index 09a9187f8aa..8e86da0eba8 100644 --- a/website/docs/docs/build/incremental-strategy.md +++ b/website/docs/docs/build/incremental-strategy.md @@ -19,22 +19,6 @@ Click the name of the adapter in the below table for more information about supp The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in dbt v1.6. - - -| data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | -|-----------------------------------------------------------------------------------------------------|:--------:|:-------:|:---------------:|:------------------:| -| [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | ✅ | | ✅ | | -| [dbt-redshift](/reference/resource-configs/redshift-configs#incremental-materialization-strategies) | ✅ | | ✅ | | -| [dbt-bigquery](/reference/resource-configs/bigquery-configs#merge-behavior-incremental-models) | | ✅ | | ✅ | -| [dbt-spark](/reference/resource-configs/spark-configs#incremental-models) | ✅ | ✅ | | ✅ | -| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) | ✅ | ✅ | | ✅ | -| [dbt-snowflake](/reference/resource-configs/snowflake-configs#merge-behavior-incremental-models) | ✅ | ✅ | ✅ | | -| [dbt-trino](/reference/resource-configs/trino-configs#incremental) | ✅ | ✅ | ✅ | | - - - - - | data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | |-----------------------------------------------------------------------------------------------------|:--------:|:-------:|:---------------:|:------------------:| | [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | ✅ | ✅ | ✅ | | @@ -46,7 +30,6 @@ The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in | [dbt-trino](/reference/resource-configs/trino-configs#incremental) | ✅ | ✅ | ✅ | | | [dbt-fabric](/reference/resource-configs/fabric-configs#incremental) | ✅ | | ✅ | | - :::note Snowflake Configurations diff --git a/website/docs/docs/build/materializations.md b/website/docs/docs/build/materializations.md index eb150a2b20c..5deb1e7ce92 100644 --- a/website/docs/docs/build/materializations.md +++ b/website/docs/docs/build/materializations.md @@ -94,7 +94,8 @@ When using the `table` materialization, your model is rebuilt as a expression. +`ephemeral` models are not directly built into the database. Instead, dbt will interpolate the code from an ephemeral model into its dependent models using a common table expression (). You can control the identifier for this CTE using a [model alias](/docs/build/custom-aliases), but dbt will always prefix the model identifier with `__dbt__cte__`. + * **Pros:** * You can still write reusable logic - Ephemeral models can help keep your clean by reducing clutter (also consider splitting your models across multiple schemas by [using custom schemas](/docs/build/custom-schemas)). diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md index 5e0772f517e..9458487e8d4 100644 --- a/website/docs/docs/build/measures.md +++ b/website/docs/docs/build/measures.md @@ -141,13 +141,13 @@ semantic_models: description: Distinct count of transactions expr: transaction_id agg: count_distinct - - name: transactions + - name: transaction_amount_avg description: The average value of transactions expr: transaction_amount_usd agg: average - name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition description: The total USD value of valid transactions only - expr: CASE WHEN is_valid = True then 1 else 0 end + expr: CASE WHEN is_valid = True then transaction_amount_usd else 0 end agg: sum - name: transactions description: The average value of transactions. diff --git a/website/docs/docs/build/metricflow-commands.md b/website/docs/docs/build/metricflow-commands.md index 09822fd70ad..c8ba951733c 100644 --- a/website/docs/docs/build/metricflow-commands.md +++ b/website/docs/docs/build/metricflow-commands.md @@ -67,7 +67,7 @@ MetricFlow provides the following commands to retrieve metadata and query metric -You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. +You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands and flags, run the `dbt sl --help` command in your terminal. - [`list`](#list) — Retrieves metadata values. - [`list metrics`](#list-metrics) — Lists metrics with dimensions. @@ -76,6 +76,9 @@ You can use the `dbt sl` prefix before the command name to execute them in the d - [`list entities`](#list-entities) — Lists all unique entities. - [`list saved queries`](#list-saved-queries) — Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query. - [`query`](#query) — Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started. +- [`export`](#export) — Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. +- [`export-all`](#export-all) — Runs exports for multiple saved queries at once, saving time and effort. + - - ### Which metrics are available? -You can define and query metrics using the [dbt Semantic Layer](/docs/build/about-metricflow), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL). To learn more, refer to [Get started with MetricFlow](/docs/build/sl-getting-started). +You can define and query metrics using the [dbt Semantic Layer](/docs/build/about-metricflow), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL).
Example query @@ -944,10 +938,6 @@ query ($environmentId: BigInt!, $first: Int!) {
-
- - - ## Governance You can use the Discovery API to audit data development and facilitate collaboration within and between teams. @@ -1041,8 +1031,6 @@ query ($environmentId: BigInt!, $first: Int!) { ``` - - ## Development You can use the Discovery API to understand dataset changes and usage and gauge impacts to inform project definition. Below are example questions and queries you can run. diff --git a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md index cb950c4fa8c..1c4d5f387e9 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md +++ b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md @@ -6,14 +6,6 @@ tags: [Semantic Layer, API] hide_table_of_contents: true pagination_next: "docs/dbt-cloud-apis/sl-jdbc" --- - - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - The rapid growth of different tools in the modern data stack has helped data professionals address the diverse needs of different teams. The downside of this growth is the fragmentation of business logic across teams, tools, and workloads.

diff --git a/website/docs/docs/dbt-cloud-apis/sl-graphql.md b/website/docs/docs/dbt-cloud-apis/sl-graphql.md index 421db45ff19..2898b6e5c0a 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-graphql.md +++ b/website/docs/docs/dbt-cloud-apis/sl-graphql.md @@ -5,15 +5,6 @@ description: "Integrate and use the GraphQL API to query your metrics." tags: [Semantic Layer, APIs] --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - - [GraphQL](https://graphql.org/) (GQL) is an open-source query language for APIs. It offers a more efficient and flexible approach compared to traditional RESTful APIs. With GraphQL, users can request specific data using a single query, reducing the need for many server round trips. This improves performance and minimizes network overhead. diff --git a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md index bb5c8fe7918..672f34b2fec 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md +++ b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md @@ -5,14 +5,6 @@ description: "Integrate and use the JDBC API to query your metrics." tags: [Semantic Layer, API] --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer Java Database Connectivity (JDBC) API enables users to query metrics and dimensions using the JDBC protocol, while also providing standard metadata functionality. A JDBC driver is a software component enabling a Java application to interact with a data platform. Here's some more information about our JDBC API: diff --git a/website/docs/docs/dbt-cloud-apis/sl-manifest.md b/website/docs/docs/dbt-cloud-apis/sl-manifest.md index 0dbbeb7b695..e203f4a0754 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-manifest.md +++ b/website/docs/docs/dbt-cloud-apis/sl-manifest.md @@ -7,14 +7,6 @@ sidebar_label: "Semantic manifest" pagination_next: null --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - dbt creates an [artifact](/reference/artifacts/dbt-artifacts) file called the _Semantic Manifest_ (`semantic_manifest.json`), which MetricFlow requires to build and run metric queries properly for the dbt Semantic Layer. This artifact contains comprehensive information about your dbt Semantic Layer. It is an internal file that acts as the integration point with MetricFlow. By using the semantic manifest produced by dbt Core, MetricFlow will instantiate a data flow plan and generate SQL from Semantic Layer query requests. It's a valuable reference that you can use to understand the structure and details of your data models. diff --git a/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md b/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md index e5d35f6b597..544590b18df 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md +++ b/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md @@ -9,7 +9,7 @@ displayed_sidebar: "docs" - Changelog (coming soon) - [dbt Core CLI Installation guide](/docs/core/installation-overview) -- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud) — dbt Cloud is now versionless. dbt v1.9 will not appear in the version dropdown. Select "Keep on latest version" to get all the latest features and functionality in your dbt Cloud account. +- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud) — dbt Cloud is now versionless. dbt v1.9 will not appear in the version dropdown. Select **Versionless** to get all the latest features and functionality in your dbt Cloud account. ## What to know before upgrading diff --git a/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md b/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md index e1e57b2e3a4..dd22329668c 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md +++ b/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md @@ -15,11 +15,11 @@ displayed_sidebar: "docs" dbt Labs is committed to providing backward compatibility for all versions 1.x, except for any changes explicitly mentioned on this page. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new). -## Keep on latest version +## Versionless dbt Cloud is going "versionless." This means you'll automatically get early access to new features and functionality before they're available in final releases of dbt Core. -Select ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) in your development, staging, and production [environments](/docs/deploy/deploy-environments) to access to everything in dbt Core v1.8 and more. +Select [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) in your development, staging, and production [environments](/docs/deploy/deploy-environments) to access to everything in dbt Core v1.8+ and more. To upgrade an environment in the [dbt Cloud Admin API](/docs/dbt-cloud-apis/admin-cloud-api) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest), set `dbt_version` to the string `versionless`. diff --git a/website/docs/docs/dbt-versions/core-versions.md b/website/docs/docs/dbt-versions/core-versions.md index f94a4c0cdf3..4a490f96bd5 100644 --- a/website/docs/docs/dbt-versions/core-versions.md +++ b/website/docs/docs/dbt-versions/core-versions.md @@ -12,7 +12,7 @@ dbt Core releases follow [semantic versioning](https://semver.org/) guidelines. _Did you know that you can always be working with the latest features and functionality?_ -With dbt Cloud, you can get early access to new functionality before it becomes available in dbt Core and without the need of managing your own version upgrades. Refer to the [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) setting for details. +With dbt Cloud, you can get early access to new functionality before it becomes available in dbt Core and without the need of managing your own version upgrades. Refer to the [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) setting for details. ::: diff --git a/website/docs/docs/dbt-versions/product-lifecycles.md b/website/docs/docs/dbt-versions/product-lifecycles.md index 313be55228a..e8711c825c4 100644 --- a/website/docs/docs/dbt-versions/product-lifecycles.md +++ b/website/docs/docs/dbt-versions/product-lifecycles.md @@ -17,7 +17,7 @@ dbt Cloud features all fall into one of the following categories: - **Beta:** Beta features are still in development and are only available to select customers. To join a beta, there might be a signup form or dbt Labs may contact specific customers about testing. Some features can be activated by enabling [experimental features](/docs/dbt-versions/experimental-features) in your account. Beta features are incomplete and might not be entirely stable; they should be used at the customer’s risk, as breaking changes could occur. Beta features might not be fully documented, technical support is limited, and service level objectives (SLOs) might not be provided. Download the [Beta Features Terms and Conditions](/assets/beta-tc.pdf) for more details. - **Preview:** Preview features are stable and considered functionally ready for production deployments. Some planned additions and modifications to feature behaviors could occur before they become generally available. New functionality that is not backward compatible could also be introduced. Preview features include documentation, technical support, and service level objectives (SLOs). Features in preview are provided at no extra cost, although they might become paid features when they become generally available. -- **Generally available (GA):** Generally available features provide stable features introduced to all qualified dbt Cloud accounts. Service level agreements (SLAs) apply to GA features, including documentation and technical support. Certain GA feature availability is determined by the dbt version of the environment. To always receive the latest GA features, go versionless and ensure your dbt Cloud [environments](/docs/dbt-cloud-environments) are set to ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). +- **Generally available (GA):** Generally available features provide stable features introduced to all qualified dbt Cloud accounts. Service level agreements (SLAs) apply to GA features, including documentation and technical support. Certain GA feature availability is determined by the dbt version of the environment. To always receive the latest GA features, ensure your dbt Cloud [environments](/docs/dbt-cloud-environments) are set to ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless). - **Deprecated:** Features in this state are no longer being developed or enhanced by dbt Labs. They will continue functioning as-is, and their documentation will persist until their removal date. However, they are no longer subject to technical support. - **Removed:** Removed features are no longer available on the platform in any capacity. diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index ca291fadbe6..b48dd344dfc 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,17 +19,27 @@ Release notes are grouped by month for both multi-tenant and virtual private clo [^*] The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## July 2024 +- **New:** [Connections](/docs/cloud/connect-data-platform/about-connections#connection-management) are now available under **Account settings** as a global setting. Previously, they were found under **Project settings**. This is being rolled out in phases over the coming weeks. +- **New:** Admins can now assign [environment-level permissions](/docs/cloud/manage-access/environment-permissions) to groups for specific roles. +- **New:** [Merge jobs](/docs/deploy/merge-jobs) for implementing [continuous deployment (CD)](/docs/deploy/continuous-deployment) workflows are now GA in dbt Cloud. Previously, you had to either set up a custom GitHub action or manually build the changes every time a pull request is merged. +- **New**: The ability to lint your SQL files from the dbt Cloud CLI is now available. To learn more, refer to [Lint SQL files](/docs/cloud/configure-cloud-cli#lint-sql-files). - **Behavior change:** dbt Cloud IDE automatically adds a `--limit 100` to preview queries to avoid slow and expensive queries during development. Recently, dbt Core changed how the `limit` is applied to ensure that `order by` clauses are consistently respected. Because of this, queries that already contain a limit clause might now cause errors in the IDE previews. To address this, dbt Labs plans to provide an option soon to disable the limit from being applied. Until then, dbt Labs recommends removing the (duplicate) limit clause from your queries during previews to avoid these IDE errors. - - **Enhancement:** Custom configurations are now supported by generic data tests. Use this to configure how dbt should run the data test (for example, specifying a Snowflake virtual warehouse different from the one in your connection). To learn more, refer to [Specify custom configurations for generic data tests](/reference/data-test-configs#specify-custom-configurations-for-generic-data-tests). - Support for this configuration is available in dbt Cloud when selecting "Keep on latest version." Specifying custom configurations for data tests will become available in dbt Core later this year. + Support for this configuration is available in dbt Cloud when selecting **Versionless**. Specifying custom configurations for data tests will become available in dbt Core later this year. - **Enhancement**: Introducing a revamped overview page for dbt Explorer, available in beta. It includes a new design and layout for the Explorer homepage. The new layout provides a more intuitive experience for users to navigate their dbt projects, as well as a new **Latest updates** section to view the latest changes or issues related to project resources. To learn more, refer to [Overview page](/docs/collaborate/explore-projects#overview-page). -- **Enhancement**: The dbt Semantic Layer now offers more granular control by supporting multiple data platform credentials, which can represent different roles or service accounts. Available for dbt Cloud Enterprise plans, you can map credentials to service tokens for secure authentication. Refer to [Set up dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) for more details. + +#### dbt Semantic Layer - **New**: Introduced the [`dbt-sl-sdk` Python software development kit (SDK)](https://github.com/dbt-labs/semantic-layer-sdk-python) Python library, which provides you with easy access to the dbt Semantic Layer with Python. It allows developers to interact with the dbt Semantic Layer APIs and query metrics and dimensions in downstream tools. Refer to the [dbt Semantic Layer Python SDK](/docs/dbt-cloud-apis/sl-python) for more information. -- **New**: The ability to lint your SQL files from the dbt Cloud CLI is now available. To learn more, refer to [Lint SQL files](/docs/cloud/configure-cloud-cli#lint-sql-files). - **New**: Introduced Semantic validations in CI pipelines. Automatically test your semantic nodes (metrics, semantic models, and saved queries) during code reviews by adding warehouse validation checks in your CI job using the `dbt sl validate` command. You can also validate modified semantic nodes to guarantee code changes made to dbt models don't break these metrics. Refer to [Semantic validations in CI](/docs/deploy/ci-jobs#semantic-validations-in-ci) to learn about the additional commands and use cases. +- **New**: We now expose the `meta` field within the [config property](/reference/resource-configs/meta) for dbt Semantic Layer metrics in the [JDBC and GraphQL APIs](/docs/dbt-cloud-apis/sl-api-overview) under the `meta` field. +- **New**: Added a new command in the dbt Cloud CLI called `export-all`, which allows you to export multiple or all of your saved queries. Previously, you had to explicitly specify the [list of saved queries](/docs/build/metricflow-commands#list-saved-queries). +- **Enhancement**: The dbt Semantic Layer now offers more granular control by supporting multiple data platform credentials, which can represent different roles or service accounts. Available for dbt Cloud Enterprise plans, you can map credentials to service tokens for secure authentication. Refer to [Set up dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) for more details. +- **Fix**: Addressed a bug where unicode query filters (such as Chinese characters) were not working correctly in the dbt Semantic Layer Tableau integration. +- **Fix**: Resolved a bug with parsing certain private keys for BigQuery when running an export. +- **Fix**: Addressed a bug that caused a "closed connection" error to be returned when querying or running an Export. +- **Fix**: Resolved an issue in dbt Core where, during partial parsing, all generated metrics in a file were incorrectly deleted instead of just those related to the changed semantic model. Now, only the metrics associated with the modified model are affected. ## June 2024 - **New:** Introduced new granularity support for cumulative metrics in MetricFlow. Granularity options for cumulative metrics are slightly different than granularity for other metric types. For other metrics, we use the `date_trunc` function to implement granularity. However, because cumulative metrics are non-additive (values can't be added up), we can't use the `date_trunc` function to change their time grain granularity. @@ -125,15 +135,15 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show - **New**: The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) introduces [declarative caching](/docs/use-dbt-semantic-layer/sl-cache), allowing you to cache common queries to speed up performance and reduce query compute costs. Available for dbt Cloud Team or Enterprise accounts. -- +- - The **Keep on latest version** setting is now Generally Available (previously Public Preview). + The **Versionless** setting is now Generally Available (previously Public Preview). - When the new **Keep on latest version** setting is enabled, you get a versionless experience and always get the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. + When the new **Versionless** setting is enabled, you get a versionless experience and always get the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. - To learn more about the new setting, refer to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) for details. + To learn more about the new setting, refer to [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) for details. - + @@ -149,7 +159,7 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show -- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and dbt Cloud "Keep on latest version"). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information. +- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and "Versionless" dbt Cloud). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information. **dbt Semantic Layer** - **New**: Use Saved selections to [save your query selections](/docs/cloud-integrations/semantic-layer/gsheets#using-saved-selections) within the [Google Sheets application](/docs/cloud-integrations/semantic-layer/gsheets). They can be made private or public and refresh upon loading. @@ -204,15 +214,15 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show -- +- _Now available in the dbt version dropdown in dbt Cloud — starting with select customers, rolling out to wider availability through February and March._ - When the new **Keep on latest version** setting is enabled, you always get the latest fixes and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. + When the new **Versionless** setting is enabled, you always get the latest fixes and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. - To learn more about the new setting, refer to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) for details. + To learn more about the new setting, refer to [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) for details. - + diff --git a/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md b/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md index 2de302993d3..0c725f129f8 100644 --- a/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md +++ b/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md @@ -7,15 +7,15 @@ In dbt Cloud, both [jobs](/docs/deploy/jobs) and [environments](/docs/dbt-cloud- ## Environments -Navigate to the settings page of an environment, then click **Edit**. Click the **dbt version** dropdown bar and make your selection. You can select a previous release of dbt Core or go versionless by selecting [**Keep on latest version**](#keep-on-latest-version)(recommended). Be sure to save your changes before navigating away. +Navigate to the settings page of an environment, then click **Edit**. Click the **dbt version** dropdown bar and make your selection. You can select a previous release of dbt Core or go [**Versionless**](#versionless)(recommended). Be sure to save your changes before navigating away. -### Keep on latest version +### Versionless -By choosing to **Keep on latest version**, you opt for a versionless experience that provides the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades for you, as part of testing and redeploying the dbt Cloud SaaS application. Keep on latest version always includes the most recent version of dbt Core, and more. +By choosing to go **Versionless**, you opt for an experience that provides the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades for you, as part of testing and redeploying the dbt Cloud SaaS application. Versionless always includes the most recent features before they're in dbt Core, and more. -You can upgrade to **Keep on latest version** and the versionless experience no matter which version of dbt you currently have selected. As a best practice, dbt Labs recommends that you test the upgrade in development first; use the [Override dbt version](#override-dbt-version) setting to test _your_ project on the latest dbt version before upgrading your deployment environments and the default development environment for all your colleagues. +You can upgrade to the **Versionless** experience no matter which version of dbt you currently have selected. As a best practice, dbt Labs recommends that you test the upgrade in development first; use the [Override dbt version](#override-dbt-version) setting to test _your_ project on the latest dbt version before upgrading your deployment environments and the default development environment for all your colleagues. To upgrade an environment in the [dbt Cloud Admin API](/docs/dbt-cloud-apis/admin-cloud-api) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest), set `dbt_version` to the string `versionless`. diff --git a/website/docs/docs/dbt-versions/versionless-cloud.md b/website/docs/docs/dbt-versions/versionless-cloud.md index ae92e0d6a0a..17a97b0f319 100644 --- a/website/docs/docs/dbt-versions/versionless-cloud.md +++ b/website/docs/docs/dbt-versions/versionless-cloud.md @@ -1,18 +1,18 @@ --- -title: "Upgrade to \"Keep on latest version\" in dbt Cloud" -sidebar_label: "Upgrade to \"Keep on latest version\" " +title: "Upgrade to \"Versionless\" in dbt Cloud" +sidebar_label: "Upgrade to \"Versionless\" " description: "Learn how to go versionless in dbt Cloud. You never have to perform an upgrade again. Plus, you'll be able to access new features and enhancements as soon as they become available. " --- -dbt Cloud is going versionless. Soon, your environments and jobs will always run on the latest version of dbt. +dbt Cloud is going versionless. Soon, your environments and jobs will always run the latest features and functionality. This will require you to make one final update to your current jobs and environments. When that's done, you'll never have to think about managing, coordinating, or upgrading dbt versions again. -Move your environments and jobs to "Keep on latest version" to get all the functionality in the latest versions of dbt Core — and more! — along with access to the new features and fixes as soon as they’re released. +Move your environments and jobs to "Versionless" to get all the functionality in the latest features before they're in dbt Core — and more! — along with access to the new features and fixes as soon as they’re released. ## Tips for upgrading {#upgrade-tips} -If you regularly develop your dbt project in dbt Cloud and this is your first time trying “Keep on latest version,” dbt Labs recommends that you start in development because it will be the fastest for investigation and iteration. [Override your dbt version in development](/docs/dbt-versions/upgrade-dbt-version-in-cloud#override-dbt-version). Then, launch the IDE or Cloud CLI and do your development work as usual. Everything should work as you expect. +If you regularly develop your dbt project in dbt Cloud and this is your first time trying “Versionless,” dbt Labs recommends that you start in development because it will be the fastest for investigation and iteration. [Override your dbt version in development](/docs/dbt-versions/upgrade-dbt-version-in-cloud#override-dbt-version). Then, launch the IDE or Cloud CLI and do your development work as usual. Everything should work as you expect. If you do see something unexpected or surprising, revert back to the previous version and record the differences you observed. [Contact dbt Cloud support](/docs/dbt-support#dbt-cloud-support) with your findings for a more detailed investigation. @@ -24,7 +24,7 @@ If your organization has multiple dbt projects, we recommend starting your upgra The following is our guidance on some important considerations regarding dbt projects as part of the upgrade. -To learn more about how dbt Labs deploys stable dbt upgrades in a safe manner to dbt Cloud, we recommend that you read our blog post [How we're making sure you can confidently "Keep on latest version" in dbt Cloud](https://docs.getdbt.com/blog/latest-dbt-stability) for details. +To learn more about how dbt Labs deploys stable dbt upgrades in a safe manner to dbt Cloud, we recommend that you read our blog post [How we're making sure you can confidently go "Versionless" in dbt Cloud](https://docs.getdbt.com/blog/latest-dbt-stability) for details. @@ -44,9 +44,9 @@ When we talk about _latest version_, we’re referring to the underlying runtime If a new version of a dbt package includes a breaking change (for example, a change to one of the macros in `dbt_utils`), you don’t have to immediately use the new version. In your `packages` configuration (in `dependencies.yml` or `packages.yml`), you can still specify which versions or version ranges of packages you want dbt to install. If you're not already doing so, we strongly recommend [checking `package-lock.yml` into version control](/reference/commands/deps#predictable-package-installs) for predictable package installs in deployment environments and a clear change history whenever you install upgrades. -If you upgrade to “Keep on latest version” and immediately see something that breaks, please [contact support](/docs/dbt-support#dbt-cloud-support) and, in the meantime, downgrade back to v1.7. +If you upgrade to “Versionless” and immediately see something that breaks, please [contact support](/docs/dbt-support#dbt-cloud-support) and, in the meantime, downgrade back to v1.7. -If you’re already on “Keep on latest version” and you observe a breaking change (like something worked yesterday, but today it isn't working, or works in a surprising/different way), please [contact support](/docs/dbt-support#dbt-cloud-support) immediately. Depending on your contracted support agreement, the dbt Labs team will respond within our SLA time and we would seek to roll back the change and/or roll out a fix (just as we would for any other part of dbt Cloud). This is the same whether or not the root cause of the breaking change is in the project code or in the code of a package. +If you’re already on “Versionless” and you observe a breaking change (like something worked yesterday, but today it isn't working, or works in a surprising/different way), please [contact support](/docs/dbt-support#dbt-cloud-support) immediately. Depending on your contracted support agreement, the dbt Labs team will respond within our SLA time and we would seek to roll back the change and/or roll out a fix (just as we would for any other part of dbt Cloud). This is the same whether or not the root cause of the breaking change is in the project code or in the code of a package. If the package you’ve installed relies on _undocumented_ functionality of dbt, it doesn't have the same guarantees as functionality that we’ve documented and tested. However, we will still do our best to avoid breaking them. @@ -54,11 +54,11 @@ If the package you’ve installed relies on _undocumented_ functionality of db -No. Going forward, “Keep on latest version” will be how all customers are going to access new functionality and ongoing support in dbt Cloud. We believe this is the best way for us to offer a reliable, stable, and secure runtime for dbt with continuous and consistent updates. +No. Going forward, “Versionless” will be how all customers are going to access new functionality and ongoing support in dbt Cloud. We believe this is the best way for us to offer a reliable, stable, and secure runtime for dbt with continuous and consistent updates. In 2023 (and earlier), customers were expected to manage their own upgrades by selecting dbt Core versions, up to and including dbt Core v1.7, which was released in October 2023. (Way back in 2021, dbt Cloud customers would pick specific _patch releases_ of dbt Core, such as upgrading from `v0.21.0` to `v0.21.1`. We’ve come a long way since then!) -In 2024, we've changed the way that new dbt functionality is made available for dbt Cloud customers: continuously. Behavior or breaking changes are gated behind opt-in flags. Users don't need to spend valuable time managing their own upgrades. This is called "Keep on latest version" and it's required for accessing any new functionality that we've put out in 2024+. +In 2024, we've changed the way that new dbt functionality is made available for dbt Cloud customers: continuously. Behavior or breaking changes are gated behind opt-in flags. Users don't need to spend valuable time managing their own upgrades. This is called "Versionless" dbt Cloud and it's required for accessing any new functionality that we've put out in 2024+. We will absolutely continue to release new minor versions of dbt Core (OSS), including v1.9 which will be available later this year. When we do, it will be a subset of the functionality that's already available to dbt Cloud customers and always after the functionality has been available in dbt Cloud. diff --git a/website/docs/docs/deploy/ci-jobs.md b/website/docs/docs/deploy/ci-jobs.md index a96311a850f..bd4b117d586 100644 --- a/website/docs/docs/deploy/ci-jobs.md +++ b/website/docs/docs/deploy/ci-jobs.md @@ -216,7 +216,11 @@ If you're on a Virtual Private dbt Enterprise plan using security features like When you start a CI job, the pull request status should show as `pending` while it waits for an update from dbt. Once the CI job finishes, dbt sends the status to Azure DevOps (ADO), and the status will change to either `succeeded` or `failed`. -If the status doesn't get updated after the job runs, check if there are any git branch policies in place that's blocking ADO from receiving these updates. You can find relevant information here: +If the status doesn't get updated after the job runs, check if there are any git branch policies in place blocking ADO from receiving these updates. + +One potential issue is the **Reset conditions** under **Status checks** in the ADO repository branch policy. If you enable the **Reset status whenever there are new changes** checkbox (under **Reset conditions**), it can prevent dbt from updating ADO about your CI job run status. +You can find relevant information here: +- [Azure DevOps Services Status checks](https://learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops&tabs=browser#status-checks) - [Azure DevOps Services Pull Request Stuck Waiting on Status Update](https://support.hashicorp.com/hc/en-us/articles/18670331556627-Azure-DevOps-Services-Pull-Request-Stuck-Waiting-on-Status-Update-from-Terraform-Cloud-Enterprise-Run) - [Pull request status](https://learn.microsoft.com/en-us/azure/devops/repos/git/pull-request-status?view=azure-devops#pull-request-status) diff --git a/website/docs/docs/deploy/continuous-deployment.md b/website/docs/docs/deploy/continuous-deployment.md new file mode 100644 index 00000000000..02144e0040e --- /dev/null +++ b/website/docs/docs/deploy/continuous-deployment.md @@ -0,0 +1,20 @@ +--- +title: "Continuous deployment in dbt Cloud" +sidebar_label: "Continuous deployment" +description: "Learn about continuous deployment (CD) workflows " +--- + +To help you improve data transformations and ship data products faster, you can run [merge jobs](/docs/deploy/merge-jobs) to implement a continuous deployment (CD) workflow in dbt Cloud. Merge jobs can automatically build modified models whenever a pull request (PR) merges, making sure the latest code changes are in production. You don't have to wait for the next scheduled job to run to get the latest updates. + +You can also implement continuous integration (CI) in dbt Cloud, which can help further to reduce the time it takes to push changes to production and improve code quality. To learn more, refer to [Continuous integration in dbt Cloud](/docs/deploy/continuous-integration). + +## How merge jobs work + +When you set up merge jobs, dbt Cloud listens for notifications from your [Git provider](/docs/cloud/git/git-configuration-in-dbt-cloud) indicating that a PR has been merged. When dbt Cloud receives one of these notifications, it enqueues a new run of the merge job. + +You can set up merge jobs to perform one of the following when a PR merges: + +|
Command to run
| Usage description | +| -------- | ----------------- | +| `dbt build --select state:modified+` | (Default) Build the modified data with every merge.

dbt Cloud builds only the changed data models and anything downstream of it, similar to CI jobs. This helps reduce computing costs and ensures that the latest code changes are always pushed to production. | +| `dbt compile` | Refresh the applied state for performant (the slimmest) CI job runs.

dbt Cloud generates the executable SQL (from the source model, test, and analysis files) but does not run it. This ensures the changes are reflected in the manifest for the next time a CI job is run and keeps track of only the relevant changes. | diff --git a/website/docs/docs/deploy/continuous-integration.md b/website/docs/docs/deploy/continuous-integration.md index bf27f68a863..fbe93e084b6 100644 --- a/website/docs/docs/deploy/continuous-integration.md +++ b/website/docs/docs/deploy/continuous-integration.md @@ -50,6 +50,6 @@ When you push a new commit to a PR, dbt Cloud enqueues a new CI run for the late -### Run slot treatment +### Run slot treatment -For accounts on the [Enterprise or Team](https://www.getdbt.com/pricing) plans, CI runs won't consume run slots. This guarantees a CI check will never block a production run. +CI runs don't consume run slots. This guarantees a CI check will never block a production run. diff --git a/website/docs/docs/deploy/deploy-environments.md b/website/docs/docs/deploy/deploy-environments.md index 50d1b7ac99e..088ecb0d841 100644 --- a/website/docs/docs/deploy/deploy-environments.md +++ b/website/docs/docs/deploy/deploy-environments.md @@ -57,13 +57,10 @@ Some customers prefer to connect Development and Staging to their `main` branch ### Why use a staging environment -There are two primary motivations for using a Staging environment: +These are the primary motivations for using a Staging environment: 1. An additional validation layer before changes are deployed into Production. You can deploy, test, and explore your dbt models in Staging. 2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments. - -:::info Coming soon: environment-level permissions -Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down. -::: +3. Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down using [environment-level permissions](/docs/cloud/manage-access/environment-permissions). **Conditional configuration of sources** enables you to point to "prod" or "non-prod" source data, depending on the environment you're running in. For example, this source will point to `.sensitive_source.table_with_pii`, where `` is dynamically resolved based on an environment variable. diff --git a/website/docs/docs/deploy/deployment-overview.md b/website/docs/docs/deploy/deployment-overview.md index 2e90fdc8d05..539b083e42e 100644 --- a/website/docs/docs/deploy/deployment-overview.md +++ b/website/docs/docs/deploy/deployment-overview.md @@ -44,9 +44,9 @@ Learn how to use dbt Cloud's features to help your team ship timely and quality icon="dbt-bit"/> - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - -
- The dbt Semantic Layer, powered by [MetricFlow](/docs/build/about-metricflow), simplifies the process of defining and using critical business metrics, like `revenue` in the modeling layer (your dbt project). By centralizing metric definitions, data teams can ensure consistent self-service access to these metrics in downstream data tools and applications. The dbt Semantic Layer eliminates duplicate coding by allowing data teams to define metrics on top of existing models and automatically handles data joins. Moving metric definitions out of the BI layer and into the modeling layer allows data teams to feel confident that different business units are working from the same metric definitions, regardless of their tool of choice. If a metric definition changes in dbt, it’s refreshed everywhere it’s invoked and creates consistency across all applications. To ensure secure access control, the dbt Semantic Layer implements robust [access permissions](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) mechanisms. diff --git a/website/docs/docs/use-dbt-semantic-layer/exports.md b/website/docs/docs/use-dbt-semantic-layer/exports.md index 2ce07746339..02480ad6617 100644 --- a/website/docs/docs/use-dbt-semantic-layer/exports.md +++ b/website/docs/docs/use-dbt-semantic-layer/exports.md @@ -58,7 +58,17 @@ There are two ways to run an export: ## Exports in development -You can run an export in your development environment using your development credentials if you want to test the output of the export before production. You can use the following command to run exports in the dbt Cloud CLI: +You can run an export in your development environment using your development credentials if you want to test the output of the export before production. + +This section explains the different commands and options available to run exports in development. + +- Use the [`dbt sl export` command](#exports-for-single-saved-query) to test and generate exports in your development environment for a singular saved query. You can also use the `--select` flag to specify particular exports from a saved query. + +- Use the [`dbt sl export-all` command](#exports-for-multiple-saved-queries) to run exports for multiple saved queries at once. This command provides a convenient way to manage and execute exports for several queries simultaneously, saving time and effort. + +### Exports for single saved query + +Use the following command to run exports in the dbt Cloud CLI: ```bash dbt sl export @@ -78,13 +88,11 @@ The following table lists the options for `dbt sl export` command, using the `-- You can also run any export defined for the saved query and write the table or view in your development environment. Refer to the following command example and output: -#### Example - ```bash dbt sl export --saved-query sq_name ``` -#### Output +The output would look something like this: ```bash Polling for export status - query_id: 2c1W6M6qGklo1LR4QqzsH7ASGFs.. @@ -119,6 +127,29 @@ dbt sl export --saved-query sq_number1 --export-as table --alias new_export ``` +### Exports for multiple saved queries + +Use the command, `dbt sl export-all`, to run exports for multiple saved queries at once. This is different from the `dbt sl export` command, which only runs exports for a singular saved query. For example, to run exports for multiple saved queries, you can use: + +```bash +dbt sl export-all +``` + +The output would look something like this: + +```bash +Exports completed: +- Created TABLE at `DBT_SL_TEST.new_customer_orders` +- Created VIEW at `DBT_SL_TEST.new_customer_orders_export_alias` +- Created TABLE at `DBT_SL_TEST.order_data_key_metrics` +- Created TABLE at `DBT_SL_TEST.weekly_revenue` + +Polling completed +``` + +The command `dbt sl export-all` provides the flexibility to manage multiple exports in a single command. + + ## Exports in production Enabling and executing exports in dbt Cloud optimizes data workflows and ensures real-time data access. It enhances efficiency and governance for smarter decisions. @@ -144,7 +175,7 @@ If exports aren't needed, you can set the value(s) to `FALSE` (`DBT_INCLUDE_SAVE - + 1. Click **Deploy** in the top navigation bar and choose **Environments**. diff --git a/website/docs/docs/use-dbt-semantic-layer/setup-sl.md b/website/docs/docs/use-dbt-semantic-layer/setup-sl.md index 2d4c4c7b8b8..03dc605d83c 100644 --- a/website/docs/docs/use-dbt-semantic-layer/setup-sl.md +++ b/website/docs/docs/use-dbt-semantic-layer/setup-sl.md @@ -8,14 +8,6 @@ tags: [Semantic Layer] With the dbt Semantic Layer, you can centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - ## Prerequisites import SetUp from '/snippets/_v2-sl-prerequisites.md'; diff --git a/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md b/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md index 5f0c50b3e43..d3b79337111 100644 --- a/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md +++ b/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md @@ -7,14 +7,6 @@ tags: [Semantic Layer] pagination_next: null --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer allows you to define metrics and use various interfaces to query them. The Semantic Layer does the heavy lifting to find where the queried data exists in your data platform and generates the SQL to make the request (including performing joins). diff --git a/website/docs/docs/use-dbt-semantic-layer/sl-cache.md b/website/docs/docs/use-dbt-semantic-layer/sl-cache.md index 49af039bc4c..99ae014939a 100644 --- a/website/docs/docs/use-dbt-semantic-layer/sl-cache.md +++ b/website/docs/docs/use-dbt-semantic-layer/sl-cache.md @@ -22,7 +22,7 @@ While you can use caching to speed up your queries and reduce compute time, know ## Prerequisites - dbt Cloud [Team or Enterprise](https://www.getdbt.com/) plan. -- dbt Cloud environments that are versionless by opting to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). +- dbt Cloud environments that are ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless). - A successful job run and [production environment](/docs/deploy/deploy-environments#set-as-production-environment). - For declarative caching, you need to have [exports](/docs/use-dbt-semantic-layer/exports) defined in your [saved queries](/docs/build/saved-queries) YAML configuration file. diff --git a/website/docs/faqs/API/rotate-token.md b/website/docs/faqs/API/rotate-token.md index d21db6f6165..8dea2d0b875 100644 --- a/website/docs/faqs/API/rotate-token.md +++ b/website/docs/faqs/API/rotate-token.md @@ -34,23 +34,21 @@ curl --location --request POST 'https://cloud.getdbt.com/api/v3/accounts/YOUR_AC * Find your `YOUR_CURRENT_TOKEN` by going to **Profile Settings** -> **API Access** and copying the API key. * Find [`YOUR_ACCESS_URL`](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan. - -For example, if `YOUR_ACCOUNT_ID` = `000`, `YOUR_USER_ID` = `123`, `YOUR_CURRENT_PAT_TOKEN` = `abcf9g`, and your `ACCESS_URL` = `cloud.getdbt.com`, then your curl request will be: +If `YOUR_USER_ID` = `123`, `YOUR_CURRENT_TOKEN` = `abcf9g`, then your curl request will be: ``` -curl --location --request POST 'https://cloud.getdbt.com/api/v3/accounts/000/users/123/apikey/' \ +curl --location --request POST 'https://YOUR_ACCESS_URL/api/v2/users/123/apikey/' \ + --header 'Authorization: Token abcf9g' ``` - 2. Find the new key in the API response or in dbt Cloud. 3. To find the new key in dbt Cloud, go to **Profile Settings** -> **API Access**. - ### dbt Cloud deployments -If your [dbt Cloud deployment](/docs/cloud/about-cloud/access-regions-ip-addresses) uses a different access URL, replace `cloud.getdbt.com` with the URL of your instance. +If your [dbt Cloud deployment](/docs/cloud/about-cloud/access-regions-ip-addresses) uses a different access URL, replace `YOUR_ACCESS_URL` with the URL of your instance. For example, if your deployment is Virtual Private dbt: diff --git a/website/docs/faqs/Accounts/find-user-id.md b/website/docs/faqs/Accounts/find-user-id.md index 7f7eca2cbba..09e3ed35a0b 100644 --- a/website/docs/faqs/Accounts/find-user-id.md +++ b/website/docs/faqs/Accounts/find-user-id.md @@ -14,4 +14,4 @@ To find your user ID in dbt Cloud, read the following steps: 3. In the address bar, the number after `/users` is your user ID. 4. Copy that number or save it somewhere safe.
-For example, if the URL is `https://cloud.getdbt.com/settings/accounts/12345/users/67891` — the user ID is `67891`

\ No newline at end of file +For example, if the URL is `https://YOUR_ACCESS_URL/settings/accounts/12345/users/67891` — the user ID is `67891`

\ No newline at end of file diff --git a/website/docs/faqs/Core/install-pip-os-prereqs.md b/website/docs/faqs/Core/install-pip-os-prereqs.md index 1eb6205512a..c8435b44f33 100644 --- a/website/docs/faqs/Core/install-pip-os-prereqs.md +++ b/website/docs/faqs/Core/install-pip-os-prereqs.md @@ -23,13 +23,7 @@ sudo yum install redhat-rpm-config gcc libffi-devel \ ### MacOS - The MacOS requires Python 3.8 or higher to successfully install and run dbt Core. - - - -The MacOS requires Python 3.7 or higher to successfully install and run dbt Core. - To check the Python version: @@ -61,15 +55,6 @@ pip install cryptography~=3.4 Windows requires Python and git to successfully install and run dbt Core. - - Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.8 or higher for Windows](https://www.python.org/downloads/windows/). - - - - - -Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.7 or higher for Windows](https://www.python.org/downloads/windows/). - For further questions, please see the [Python compatibility FAQ](/faqs/Core/install-python-compatibility) diff --git a/website/docs/faqs/Git/gitignore.md b/website/docs/faqs/Git/gitignore.md index 8d966c40e2c..16575861289 100644 --- a/website/docs/faqs/Git/gitignore.md +++ b/website/docs/faqs/Git/gitignore.md @@ -13,7 +13,6 @@ If you encounter issues like problems reverting changes, checking out or creatin To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. - 1. Launch the Cloud IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. @@ -121,6 +120,4 @@ dbt_modules/ - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/faqs/Models/unique-model-names.md b/website/docs/faqs/Models/unique-model-names.md index 7878a5a704c..6d8bd18ac00 100644 --- a/website/docs/faqs/Models/unique-model-names.md +++ b/website/docs/faqs/Models/unique-model-names.md @@ -6,20 +6,8 @@ id: unique-model-names --- - - Within one project: yes! To build dependencies between models, you need to use the `ref` function, and pass in the model name as an argument. dbt uses that model name to uniquely resolve the `ref` to a specific model. As a result, these model names need to be unique, _even if they are in distinct folders_. A model in one project can have the same name as a model in another project (installed as a dependency). dbt uses the project name to uniquely identify each model. We call this "namespacing." If you `ref` a model with a duplicated name, it will resolve to the model within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](/reference/dbt-jinja-functions/ref#ref-project-specific-models) to disambiguate references by specifying the namespace. Those models will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](/docs/build/custom-aliases) and [custom schemas](/docs/build/custom-schemas) for details on how to achieve this. - - - - - -Yes! To build dependencies between models, you need to use the `ref` function, and pass in the model name as an argument. dbt uses that model name to uniquely resolve the `ref` to a specific model. As a result, these model names need to be unique, _even if they are in distinct folders_. - -Often, this question comes up because users want to give two models the same name in their warehouse, splitting them across separate schemas (e.g. `stripe.users` and `app.users`). Checkout the docs on [custom aliases](/docs/build/custom-aliases) and [custom schemas](/docs/build/custom-schemas) to achieve this. - - diff --git a/website/docs/faqs/Project/why-version-2.md b/website/docs/faqs/Project/why-version-2.md index b4e91d6a773..6318ccf037d 100644 --- a/website/docs/faqs/Project/why-version-2.md +++ b/website/docs/faqs/Project/why-version-2.md @@ -6,11 +6,7 @@ id: why-version-2 --- - - Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. Resource yml files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update yml files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future - - diff --git a/website/docs/faqs/Troubleshooting/gitignore.md b/website/docs/faqs/Troubleshooting/gitignore.md index 6ab217ebf07..1519ef99ee3 100644 --- a/website/docs/faqs/Troubleshooting/gitignore.md +++ b/website/docs/faqs/Troubleshooting/gitignore.md @@ -11,8 +11,6 @@ If you can't revert changes, check out a branch, or click commit — this is To fix this, complete the following steps: - - 1. In the dbt Cloud IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: ```bash target/ @@ -37,7 +35,4 @@ dbt_modules/ - - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/faqs/Troubleshooting/job-memory-limits.md b/website/docs/faqs/Troubleshooting/job-memory-limits.md index 377651dae59..06f6a752507 100644 --- a/website/docs/faqs/Troubleshooting/job-memory-limits.md +++ b/website/docs/faqs/Troubleshooting/job-memory-limits.md @@ -15,13 +15,14 @@ Some common reasons for higher memory usage are: ## Resolution -There are various reasons why you could be experiencing this error. We recommend you review your data models to see if there are any opportunities to optimize or refactor them. For example, you can try to reduce the number of columns being selected, use `group` or `where` clauses to filter data early in the query, or use `limit` clauses to reduce the amount of data being processed. +There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. - + If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! diff --git a/website/docs/guides/adapter-creation.md b/website/docs/guides/adapter-creation.md index 4e8479594c2..b6528495260 100644 --- a/website/docs/guides/adapter-creation.md +++ b/website/docs/guides/adapter-creation.md @@ -76,12 +76,12 @@ Differences between databases are encoded into discrete areas: | Components | Code Path | Function | |------------------|---------------------------------------------------|-------------------------------------------------------------------------------| -| Python Classes | `adapters/` | Configuration (See above [Python classes](##python classes) | +| Python classes | `adapters/` | Configuration (Refer to [Python classes](#python classes) | | Macros | `include//macros/adapters/` | SQL API & statement syntax (for example, how to create schema or how to get table info) | | Materializations | `include//macros/materializations/` | Table/view/snapshot/ workflow definitions | -#### Python Classes +#### Python classes These classes implement all the methods responsible for: - Connecting to a database and issuing queries. diff --git a/website/docs/guides/airflow-and-dbt-cloud.md b/website/docs/guides/airflow-and-dbt-cloud.md index 0f1ea0f425f..51ac7668aa9 100644 --- a/website/docs/guides/airflow-and-dbt-cloud.md +++ b/website/docs/guides/airflow-and-dbt-cloud.md @@ -117,7 +117,7 @@ cd airflow-dbt-cloud - Once you hit `save` on the job, make sure you copy the URL and save it for referencing later. The url will look similar to this: ```html -https://cloud.getdbt.com/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ +https://YOUR_ACCESS_URL/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ ``` @@ -134,7 +134,7 @@ Now you have all the working pieces to get up and running with Airflow + dbt Clo ![Connection type](/img/guides/orchestration/airflow-and-dbt-cloud/connection-type.png) -3. Add in your connection details and your default dbt Cloud account id. This is found in your dbt Cloud URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/` +3. Add in your connection details and your default dbt Cloud account id. This is found in your dbt Cloud URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/` ![Connection type](/img/guides/orchestration/airflow-and-dbt-cloud/connection-type-configured.png) @@ -145,7 +145,7 @@ Now you have all the working pieces to get up and running with Airflow + dbt Clo Both IDs are included inside of the dbt Cloud job URL as shown in the following snippets: ```python -# For the dbt Cloud Job URL https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/ +# For the dbt Cloud Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The account_id is 16173 # Update line 28 @@ -153,7 +153,7 @@ default_args={"dbt_cloud_conn_id": "dbt_cloud", "account_id": 16173}, ``` ```python -# For the dbt Cloud Job URL https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/ +# For the dbt Cloud Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The job_id is 65767 # Update line 39 diff --git a/website/docs/guides/bigquery-qs.md b/website/docs/guides/bigquery-qs.md index 1ba5f7b0021..e608efeffc7 100644 --- a/website/docs/guides/bigquery-qs.md +++ b/website/docs/guides/bigquery-qs.md @@ -85,7 +85,7 @@ In order to let dbt connect to your warehouse, you'll need to generate a keyfile 3. Create a service account key for your new project from the [Service accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts?walkthrough_id=iam--create-service-account-keys&start_index=1#step_index=1). For more information, refer to [Create a service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating) in the Google Cloud docs. When downloading the JSON file, make sure to use a filename you can easily remember. For example, `dbt-user-creds.json`. For security reasons, dbt Labs recommends that you protect this JSON file like you would your identity credentials; for example, don't check the JSON file into your version control software. ## Connect dbt Cloud to BigQuery​ -1. Create a new project in [dbt Cloud](https://cloud.getdbt.com/). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. +1. Create a new project in [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. 2. Enter a project name and click **Continue**. 3. For the warehouse, click **BigQuery** then **Next** to set up your connection. 4. Click **Upload a Service Account JSON File** in settings. diff --git a/website/docs/guides/core-cloud-2.md b/website/docs/guides/core-cloud-2.md index 335b164d988..93e9e92bfa4 100644 --- a/website/docs/guides/core-cloud-2.md +++ b/website/docs/guides/core-cloud-2.md @@ -35,7 +35,7 @@ The guide outlines the following steps: ## Considerations -If your team has is using dbt Core today, you could be reading this guide because: +If your team is using dbt Core today, you could be reading this guide because: - You’ve realized the burden of maintaining that deployment. - The person who set it up has since left. - You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. @@ -46,13 +46,13 @@ The most important things you need to think about when moving from dbt Core to d - How is your team structured? Are there natural divisions of domain? - Should you have one project or multiple? Which dbt resources do you want to standardize & keep central? -- Who should have permissions to view, develop, administer? +- Who should have permission to view, develop, and administer? - How are you scheduling your dbt models to run in production? - How are you currently managing Continuous integration/Continuous deployment (CI/CD) of logical changes (if at all)? - How do your data developers prefer to work? - How do you manage different data environments and the different behaviors in those environments? -dbt Cloud provides standard mechanisms for tackling these considerations, all of which delivers long-term benefits to your organization: +dbt Cloud provides standard mechanisms for tackling these considerations, all of which deliver long-term benefits to your organization: - Cross-team collaboration - Access control - Orchestration @@ -141,7 +141,7 @@ After [setting the foundations of dbt Cloud](https://docs.getdbt.com/guides/core Once you’ve confirmed that dbt Cloud orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you’re still using an external orchestrator (such as Airflow), and you’ve swapped out `dbt-core` execution for dbt Cloud execution (through the [API](/docs/dbt-cloud-apis/overview)). Familiarize your team with dbt Cloud's [features](/docs/cloud/about-cloud/dbt-cloud-features) and optimize development and deployment processes. Some key features to consider include: -- **Version management:** Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, removing the hassle of manual updates and version discrepancies. You can go versionless by opting to **[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)** to always get the latest features and early access to new functionality for your dbt project. +- **Version management:** Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, removing the hassle of manual updates and version discrepancies. You can go [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest features and early access to new functionality for your dbt project. - **Development tools**: Use the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to build, test, run, and version control your dbt projects. - **Documentation and Source freshness:** Automate storage of [documentation](/docs/build/documentation) and track [source freshness](/docs/deploy/source-freshness) in dbt Cloud, which streamlines project maintenance. - **Notifications and logs:** Receive immediate [notifications](/docs/deploy/monitor-jobs) for job failures, with direct links to the job details. Access comprehensive logs for all job runs to help with troubleshooting. diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md index 6e130d3a29f..171e844d7e5 100644 --- a/website/docs/guides/core-to-cloud-1.md +++ b/website/docs/guides/core-to-cloud-1.md @@ -57,7 +57,7 @@ This guide outlines the steps you need to take to move from dbt Core to dbt Clou ## Prerequisites - You have an existing dbt Core project connected to a Git repository and data platform supported in [dbt Cloud](/docs/cloud/connect-data-platform/about-connections). -- A [supported version](/docs/dbt-versions/core) of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt. +- A [supported version](/docs/dbt-versions/core) of dbt or select [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) of dbt. - You have a dbt Cloud account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**! ## Account setup @@ -144,7 +144,7 @@ The most common data environments are production, staging, and development. The 1. **Set up development environment** — Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing. 2. **dbt Core version** — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run `dbt --version` in the command line to find out which version of dbt Core you’re using. - - When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) so you don’t have to. + - When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) so you don’t have to. 3. **Connect to your data platform** — When using dbt Cloud, you can [connect to your data platform](/docs/cloud/connect-data-platform/about-connections) directly in the UI. - Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project. @@ -171,9 +171,9 @@ In dbt Cloud, you can set [environment variables](/docs/build/environment-variab In dbt Core, environment variables, or the [`env_var` function](/reference/dbt-jinja-functions/env_var), are defined manually by the developer or within the external application running dbt. ### Environment variables in dbt Cloud - - dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET`). + - dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`). - If your dbt Core environment variables don’t follow this naming convention, perform a [“find and replace”](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#dbt-cloud-ide-features) in your project to make sure all references to these environment variables contain the proper naming conventions. -- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET_``DBT_ENV_SECRET` to obscure them in logs and the UI. +- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET`to obscure them in logs and the UI. @@ -206,7 +206,7 @@ To use the [dbt Cloud's job scheduler](/docs/deploy/job-scheduler), set up one e ### Initial setup steps 1. **dbt Core version** — In your environment settings, configure dbt Cloud with the same dbt Core version. - - Once your full migration is complete, we recommend upgrading your environments to a versionless experience by opting to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) to always get the latest features and more. You only need to do this once. + - Once your full migration is complete, we recommend upgrading your environments to ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest features and more. You only need to do this once. 2. **Configure your jobs** — [Create jobs](/docs/deploy/deploy-jobs#create-and-schedule-jobs) for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job. - Note that alongside [jobs in dbt Cloud](/docs/deploy/jobs), discover other ways to schedule and run your dbt jobs with the help of other tools. Refer to [Integrate with other tools](/docs/deploy/deployment-tools) for more information. diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md index cf3cd58d0d1..0ea22de8478 100644 --- a/website/docs/guides/core-to-cloud-3.md +++ b/website/docs/guides/core-to-cloud-3.md @@ -36,7 +36,7 @@ You may have already started your move to dbt Cloud and are looking for tips to In dbt Cloud, you can natively connect to your data platform and test its [connection](/docs/connect-adapters) with a click of a button. This is especially useful for users who are new to dbt Cloud or are looking to streamline their connection setup. Here are some tips and caveats to consider: ### Tips -- Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, eliminating the need for manual updates and version discrepancies. You can go versionless and **[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)** to always get the latest features and early access to new functionality for your dbt project. +- Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, eliminating the need for manual updates and version discrepancies. You can go [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest features and early access to new functionality for your dbt project. - dbt Cloud supports a whole host of [cloud providers](/docs/cloud/connect-data-platform/about-connections), including Snowflake, Databricks, BigQuery, Fabric, and Redshift (to name a few). - Use [Extended Attributes](/docs/deploy/deploy-environments#extended-attributes) to set a flexible [profiles.yml](/docs/core/connect-data-platform/profiles.yml) snippet in your dbt Cloud environment settings. It gives you more control over environments (both deployment and development) and extends how dbt Cloud connects to the data platform within a given environment. - For example, if you have a field in your `profiles.yml` that you’d like to add to the dbt Cloud adapter user interface, you can use Extended Attributes to set it. diff --git a/website/docs/guides/customize-schema-alias.md b/website/docs/guides/customize-schema-alias.md new file mode 100644 index 00000000000..28d4aada525 --- /dev/null +++ b/website/docs/guides/customize-schema-alias.md @@ -0,0 +1,545 @@ +--- +title: Customize dbt models database, schema, and alias +id: customize-schema-alias +description: "Learn how to properly adjust your generate_schema_name() and generate_alias_name() macros." +displayText: Learn how to adjust your generate schema name and generate alias name. +hoverSnippet: Learn how to adjust your generate schema name and generate alias name. +# time_to_complete: '30 minutes' commenting out until we test +icon: 'guides' +hide_table_of_contents: true +level: 'Advanced' +recently_updated: true +--- + +
+ +## Introduction +This guide explains how to customize the [schema](/docs/build/custom-schemas), [database](/docs/build/custom-databases), and [alias](/docs/build/custom-aliases) naming conventions in dbt to fit your data warehouse governance and design needs. +When we develop dbt models and execute certain [commands](https://docs.getdbt.com/reference/dbt-commands) (such as `dbt run` or `dbt build`), objects (like tables and views) get created in the data warehouse based on these naming conventions. + + + +:::info A word on naming + +Different warehouses have different names for _logical databases_. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. + +::: + + +The following is dbt's out-of-the-box default behavior: + +- The database where the object is created is defined by the database configured at the [environment level in dbt Cloud](/docs/dbt-cloud-environments) or in the [`profiles.yml` file](/docs/core/connect-data-platform/profiles.yml) in dbt Core. + +- The schema depends on whether you have defined a [custom schema](/docs/build/custom-schemas) for the model: + - If you haven't defined a custom schema, dbt creates the object in the default schema. In dbt Cloud this is typically `dbt_username` for development and the default schema for deployment environments. In dbt Core, it uses the schema specified in the `profiles.yml` file. + - If you define a custom schema, dbt concatenates the schema mentioned earlier with the custom one. + - For example, if the configured schema is `dbt_myschema` and the custom one is `marketing`, the objects will be created under `dbt_myschema_marketing`. + - Note that for automated CI jobs, the schema name derives from the job number and PR number: `dbt_cloud_pr__`. + + +- The object name depends on whether an [alias](/reference/resource-configs/alias) has been defined on the model: + - If no alias is defined, the object will be created with the same name as the model, without the `.sql` or `.py` at the end. + - For example, suppose that we have a model where the sql file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and no custom alias is configured. The resulting model will be created in `dbt_myschema_marketing.fct_orders_complete` in the dev environment. + - If an alias is defined, the object will be created with the configured alias. + - For example, suppose that we have a model where the sql file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and the alias is configured to be `fct_orders`. The resulting model will be created in `dbt_myschema_marketing.fct_orders` + +These default rules are a great starting point, and many organizations choose to stick with those without any customization required. + +The defaults allow developers to work in their isolated schemas (sandboxes) without overwriting each other's work — even if they're working on the same tables. + + +## How to customize this behavior + +While the default behavior will fit the needs of most organizations, there are occasions where this approach won't work. + +For example, dbt expects that it has permission to create schemas as needed (and we recommend that the users running dbt have this ability), but it might not be allowed at your company. + +Or, based on how you've designed your warehouse, you may wish to minimize the number of schemas in your dev environment (and avoid schema sprawl by not creating the combination of all developer schemas and custom schemas). + +Alternatively, you may even want your dev schemas to be named after feature branches instead of the developer name. + +For this reason, dbt offers three macros to customize what objects are created in the data warehouse: + +- [`generate_database_name()`](/docs/build/custom-databases#generate_database_name) +- [`generate_schema_name()`](/docs/build/custom-schemas#how-does-dbt-generate-a-models-schema-name) +- [`generate_alias_name()`](/docs/build/custom-aliases#generate_alias_name) + +By overwriting one or multiple of those macros, we can tailor where dbt objects are created in the data warehouse and align with any existing requirement. + + +:::note Key concept + +Models run from two different contexts must result in unique objects in the data warehouse. For example, a developer named Suzie is working on enhancements to `fct_player_stats`, but Darren is developing against the exact same object. + +In order to prevent overwriting each other's work, both Suzie and Darren should each have their unique versions of `fct_player_stats` in the development environment. + +Further, the staging version of `fct_player_stats` should exist in a unique location apart from the development versions, and the production version. + +::: + + +We often leverage the following when customizing these macros: + +- In dbt Cloud, we recommend utilizing [environment variables](/docs/build/environment-variables) to define where the dbt invocation is occurring (dev/stg/prod). + - They can be set at the environment level and all jobs will automatically inherit the default values. We'll add jinja logic (`if/else/endif`) to identify whether the run happens in dev, prod, Ci, and more. + +- Or as an alternative to environment variables, you can use `target.name`. For more information, you can refer to [About target variables](/reference/dbt-jinja-functions/target). + + + + +To allow the database/schema/object name to depend on the current branch, you can use the out of the box `DBT_CLOUD_GIT_BRANCH` environment variable in dbt Cloud [special environment variables](/docs/build/environment-variables#special-environment-variables). + + +## Example use cases + +Here are some typical examples we've encountered with dbt users leveraging those 3 macros and different logic. + + +:::note + +Note that the following examples are not comprehensive and do not cover all the available options. These examples are meant to be templates for you to develop your own behaviors. + +::: + + +- [Use custom schema without concatenating target schema in production](/guides/customize-schema-alias?step=3#1-custom-schemas-without-target-schema-concatenation-in-production) +- [Add developer identities to tables](/guides/customize-schema-alias?step=3#2-static-schemas-add-developer-identities-to-tables) +- [Use branch name as schema prefix](/guides/customize-schema-alias?step=3#3-use-branch-name-as-schema-prefix) +- [Use a static schema for CI](/guides/customize-schema-alias?step=3#4-use-a-static-schema-for-ci) + + +### 1. Custom schemas without target schema concatenation in production + + +The most common use case is using the custom schema without concatenating it with the default schema name when in production. + +To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: + + + + + +```jinja +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- elif env_var('DBT_ENV_TYPE','DEV') == 'PROD' -%} + + {{ custom_schema_name | trim }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Developer 1 | dev | dbt_dev1 |dev.dbt_dev1_marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.dbt_dev2_marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model| +| CI PR 234 | ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model| +| Production | prod | analytics |prod.marketing.my_model | + + +:::note + +We added logic to check if the current dbt run is happening in production or not. This is important, and we explain why in the [What not to do](/guides/customize-schema-alias?step=3#what-not-to-do) section. + +::: + + +### 2. Static schemas: Add developer identities to tables + +Occasionally, we run into instances where the security posture of the organization prevents developers from creating schemas and all developers have to develop in a single schema. + +In this case, we can: + +- Create a new file called generate_schema_name.sql under your macros folder with the following code: + +- Change `generate_schema_name()` to use a single schema for all developers, even if a custom schema is set. +- Update `generate_alias_name()` to append the developer alias and the custom schema to the front of the table name in the dev environment. + - This method is not ideal, as it can cause long table names, but it will let developers see in which schema the model will be created in production. + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- elif env_var('DBT_ENV_TYPE','DEV') != 'CI' -%} + + {{ custom_schema_name | trim }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + + + +```jinja + +{% macro generate_alias_name(custom_alias_name=none, node=none) -%} + + {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} + + {%- if custom_alias_name -%} + + {{ target.schema }}__{{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ target.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} + + {%- else -%} + + {{ target.schema }}__{{ node.name }} + + {%- endif -%} + + {%- else -%} + + {%- if custom_alias_name -%} + + {{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} + + {%- else -%} + + {{ node.name }} + + {%- endif -%} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Developer 1 | dev | dbt_dev1 |dev.marketing.dbt_dev1_my_model | +| Developer 2 | dev | dbt_dev2 |dev.marketing.dbt_dev2_my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model| +| CI PR 234 | ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model| +| Production | prod | analytics |prod.marketing.my_model | + + +### 3. Use branch name as schema prefix + +For teams who prefer to isolate work based on the feature branch, you may want to take advantage of the `DBT_CLOUD_GIT_BRANCH` special environment variable. Please note that developers will write to the exact same schema when they are on the same feature branch. + + +:::note + +The `DBT_CLOUD_GIT_BRANCH` variable is only available within the dbt Cloud IDE and not the Cloud CLI. + +::: + + +We’ve also seen some organizations prefer to organize their dev databases by branch name. This requires implementing similar logic in `generate_database_name()` instead of the `generate_schema_name()` macro. By default, dbt will not automatically create the databases. + +Refer to the [Tips and tricks](https://docs.getdbt.com/guides/customize-schema-alias?step=5) section to learn more. + + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} + + {#- we replace characters not allowed in the schema names by "_" -#} + {%- set re = modules.re -%} + {%- set cleaned_branch = re.sub("\W", "_", env_var('DBT_CLOUD_GIT_BRANCH')) -%} + + {%- if custom_schema_name is none -%} + + {{ cleaned_branch }} + + {%- else -%} + + {{ cleaned_branch }}_{{ custom_schema_name | trim }} + + {%- endif -%} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Branch |Target database| Target schema | Resulting object | +|-------------|:----------:|:-------------:|:-------------:|:---------------------------------:| +| Developer 1 |`featureABC`|dev | dbt_dev1 |dev.featureABC_marketing.my_model | +| Developer 2 |`featureABC`|dev | dbt_dev2 |dev.featureABC_marketing.my_model | +| Developer 1 |`feature123`|dev | dbt_dev1 |dev.feature123_marketing.my_model | +| CI PR 123 | |ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model | +| CI PR 234 | |ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model | +| Production | |prod | analytics |prod.marketing.my_model | + + +When developer 1 and developer 2 are checked out on the same branch, they will generate the same object in the data warehouse. This shouldn't be a problem as being on the same branch means the model's code will be the same for both developers. + + +### 4. Use a static schema for CI + +Some organizations prefer to write their CI jobs to a single schema with the PR identifier prefixed to the front of the table name. It's important to note that this will result in long table names. + +To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: + + + + +```jinja + +{% macro generate_schema_name(custom_schema_name=none, node=none) -%} + + {%- set default_schema = target.schema -%} + + {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} + {# {%- if target.name == 'CI' -%} #} + + {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} + + ci_schema + + {%- elif custom_schema_name is none -%} + + {{ default_schema }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + + + +```jinja + +{% macro generate_alias_name(custom_alias_name=none, node=none) -%} + + {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} + {# {%- if target.name == 'CI' -%} #} + {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} + + {%- if custom_alias_name -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} + + {%- else -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ node.name }} + + {%- endif -%} + + {%- else -%} + + {%- if custom_alias_name -%} + + {{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} + + {%- else -%} + + {{ node.name }} + + {%- endif -%} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:----------------------------------------: | +| Developer 1 | dev | dbt_dev1 |dev.dbt_dev1_marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.dbt_dev2_marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.ci_schema.dbt_pr_123_marketing_my_model | +| CI PR 234 | ci | dbt_pr_234 |ci.ci_schema.dbt_pr_234_marketing_my_model | +| Production | prod | analytics |prod.marketing.my_model | + + +## What not to do + +This section will provide an outline of what users should avoid doing when customizing their schema and alias due to the issues that may arise. + + +### Update generate_schema_name() to always use the custom schema + + +Some people prefer to only use the custom schema when it is set instead of concatenating the default schema with the custom one, as it happens in the out of the box behavior. + + +### Problem + +When modifying the default macro for `generate_schema_name()`, this might result in creating this new version. + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- else -%} + # The following is incorrect as it omits {{ default_schema }} before {{ custom_schema_name | trim }}. + {{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +While it may provide the expected output for production, where a dedicated database is used, it will generate conflicts anywhere people share a database. + + +Let’s look at the example of a model called `my_model` with a custom schema of `marketing`. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Production | prod | analytics |prod.marketing.my_model | +| Developer 1 | dev | dbt_dev1 |dev.marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.marketing.my_model | +| CI PR 234 | ci | dbt_pr_234 |ci.marketing.my_model | + + + +We can see that both developer 1 and developer 2 get the same object for `my_model`. This means that if they both work on this model at the same time, it will be impossible to know if the version currently in the data warehouse is the one from developer 1 and developer 2. + +Similarly, different PRs will result in the exact same object in the data warehouse. If different PRs are open at the same time and modifying the same models, it is very likely that we will get issues, slowing down the whole development and code promotion. + + +### Solution + +As described in the previous example, update the macro to check if dbt is running in production. Only in production should we remove the concatenation and use the custom schema alone. + + +## Tips and tricks + +This section will provide some useful tips on how to properly adjust your `generate_database_name()` and `generate_alias_name()` macros. + + +### Creating non existing databases from dbt + +dbt will automatically try to create a schema if it doesn’t exist and if an object needs to be created in it, but it won’t automatically try to create a database that doesn’t exist. + +So, if your `generate_database_name()` configuration points to different databases, which might not exist, dbt will fail if you do a simple `dbt build`. + +It is still possible to get it working in dbt by creating some macros that will check if a database exists and if not, dbt will create it. You can then call those macros either in [a `dbt run-operation ...` step](/reference/commands/run-operation) in your jobs or as a [`on-run-start` hook](/reference/project-configs/on-run-start-on-run-end). + + +### Assuming context using environment variables rather than `target.name` + + +We prefer to use [environment variables](/docs/build/environment-variables) over `target.name` For a further read, have a look at ([About target variables](/reference/dbt-jinja-functions/target)) to decipher the context of the dbt invocation. + +- `target.name` cannot be set at the environment-level. Therefore, every job within the environment must explicitly specify the `target.name` override. If the job does not have the appropriate `target.name` value set, the database/schema/alias may not resolve properly. Alternatively, environment variable values are inherited by the jobs within their corresponding environment. The environment variable values can also be overwritten within the jobs if needed. + + + + + +- `target.name` requires every developer to input the same value (often ‘dev’) into the target name section of their project development credentials. If a developer doesn’t have the appropriate target name value set, their database/schema/alias may not resolve properly. + + + + + +### Always enforce custom schemas + +Some users prefer to enforce custom schemas on all objects within their projects. This avoids writing to unintended “default” locations. You can add this logic to your `generate_schema_name()` macro to [raise a compilation error](/reference/dbt-jinja-functions/exceptions) if a custom schema is not defined for an object. + + + + +```jinja + + {% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none and node.resource_type == 'model' -%} + + {{ exceptions.raise_compiler_error("Error: No Custom Schema Defined for the model " ~ node.name ) }} + + {%- endif -%} + +``` + + +
diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index b969786b384..bb248e09320 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -169,16 +169,63 @@ If you get a session error and don’t get redirected to this page, you can go b There are two ways to connect dbt Cloud to Databricks. The first option is Partner Connect, which provides a streamlined setup to create your dbt Cloud account from within your new Databricks trial account. The second option is to create your dbt Cloud account separately and build the Databricks connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt Cloud setup flow, dbt Labs recommends connecting manually. -If you want to use Partner Connect, refer to [Connect to dbt Cloud using Partner Connect](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-using-partner-connect) in the Databricks docs for instructions. +## Set up the integration from Partner Connect -If you want to connect manually, refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. +:::note + Partner Connect is intended for trial partner accounts. If your organization already has a dbt Cloud account, connect manually. Refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. +::: + +To connect dbt Cloud to Databricks using Partner Connect, do the following: + +1. In the sidebar of your Databricks account, click **Partner Connect**. + +2. Click the **dbt tile**. + +3. Select a catalog from the drop-down list, and then click **Next**. The drop-down list displays catalogs you have read and write access to. If your workspace isn't `-enabled`, the legacy Hive metastore (`hive_metastore`) is used. + +5. If there are SQL warehouses in your workspace, select a SQL warehouse from the drop-down list. If your SQL warehouse is stopped, click **Start**. + +6. If there are no SQL warehouses in your workspace: + + 1. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. + 2. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. + 3. Return to the Partner Connect tab in your browser, and then close the **dbt tile**. + 4. Re-open the **dbt tile**. + 5. Select the SQL warehouse you just created from the drop-down list. + +7. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you have read and write access to. You can repeat this step to add multiple schemas. -## Set up a dbt Cloud managed repository -If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as the Partner Connect provides you with a managed repository. Otherwise, you will need to create your repository connection. + Partner Connect creates the following resources in your workspace: + + - A Databricks service principal named **DBT_CLOUD_USER**. + - A Databricks personal access token that is associated with the **DBT_CLOUD_USER** service principal. + + Partner Connect also grants the following privileges to the **DBT_CLOUD_USER** service principal: + + - (Unity Catalog) **USE CATALOG**: Required to interact with objects within the selected catalog. + - (Unity Catalog) **USE SCHEMA**: Required to interact with objects within the selected schema. + - (Unity Catalog) **CREATE SCHEMA**: Grants the ability to create schemas in the selected catalog. + - (Hive metastore) **USAGE**: Required to grant the **SELECT** and **READ_METADATA** privileges for the schemas you selected. + - **SELECT**: Grants the ability to read the schemas you selected. + - (Hive metastore) **READ_METADATA**: Grants the ability to read metadata for the schemas you selected. + - **CAN_USE**: Grants permissions to use the SQL warehouse you selected. + +8. Click **Next**. + + The **Email** box displays the email address for your Databricks account. dbt Labs uses this email address to prompt you to create a trial dbt Cloud account. + +9. Click **Connect to dbt Cloud**. + + A new tab opens in your web browser, which displays the getdbt.com website. + +10. Complete the on-screen instructions on the getdbt.com website to create your trial dbt Cloud account. + +## Set up a dbt Cloud managed repository ## Initialize your dbt project​ and start developing + Now that you have a repository configured, you can initialize your project and start development in dbt Cloud: 1. Click **Start developing in the IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. diff --git a/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md b/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md index b4cea114f1a..f420b7845a2 100644 --- a/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md +++ b/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md @@ -130,7 +130,7 @@ if __name__ == '__main__': 4. Replace **``** and **``** with the correct values of your environment and [Access URL](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan. - * To find these values, navigate to **dbt Cloud**, select **Deploy -> Jobs**. Select the Job you want to run and copy the URL. For example: `https://cloud.getdbt.com/deploy/000000/projects/111111/jobs/222222` + * To find these values, navigate to **dbt Cloud**, select **Deploy -> Jobs**. Select the Job you want to run and copy the URL. For example: `https://YOUR_ACCESS_URL/deploy/000000/projects/111111/jobs/222222` and therefore valid code would be: Your URL is structured `https:///deploy//projects//jobs/` diff --git a/website/docs/guides/mesh-qs.md b/website/docs/guides/mesh-qs.md index fe581471da3..f2e02437819 100644 --- a/website/docs/guides/mesh-qs.md +++ b/website/docs/guides/mesh-qs.md @@ -40,7 +40,7 @@ To leverage dbt Mesh, you need the following: - You must have a [dbt Cloud Enterprise account](https://www.getdbt.com/get-started/enterprise-contact-pricing) - You have access to a cloud data platform, permissions to load the sample data tables, and dbt Cloud permissions to create new projects. -- Set your development and deployment [environments](/docs/dbt-cloud-environments) to use dbt [version](/docs/dbt-versions/core) 1.6 or later. You can also opt to go versionless and select [Keep on latest version of](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) to always get the most recent features and functionality. +- Set your development and deployment [environments](/docs/dbt-cloud-environments) to use dbt [version](/docs/dbt-versions/core) 1.6 or later. You can also opt to go ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the most recent features and functionality. - This guide uses the Jaffle Shop sample data, including `customers`, `orders`, and `payments` tables. Follow the provided instructions to load this data into your respective data platform: - [Snowflake](https://docs.getdbt.com/guides/snowflake?step=3) - [Databricks](https://docs.getdbt.com/guides/databricks?step=3) diff --git a/website/docs/guides/redshift-qs.md b/website/docs/guides/redshift-qs.md index e3685595804..544c18a75d5 100644 --- a/website/docs/guides/redshift-qs.md +++ b/website/docs/guides/redshift-qs.md @@ -166,7 +166,7 @@ Now we are going to load our sample data into the S3 bucket that our Cloudformat select * from stripe.payment; ``` ## Connect dbt Cloud to Redshift -1. Create a new project in [dbt Cloud](https://cloud.getdbt.com/). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. +1. Create a new project in [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. 2. Enter a project name and click **Continue**. 3. For the warehouse, click **Redshift** then **Next** to set up your connection. 4. Enter your Redshift settings. Reference your credentials you saved from the CloudFormation template. diff --git a/website/docs/guides/sl-snowflake-qs.md b/website/docs/guides/sl-snowflake-qs.md index b80f6706c80..6226634fe42 100644 --- a/website/docs/guides/sl-snowflake-qs.md +++ b/website/docs/guides/sl-snowflake-qs.md @@ -22,14 +22,6 @@ import ConnectQueryAPI from '/snippets/_sl-connect-and-query-api.md'; import RunProdJob from '/snippets/_sl-run-prod-job.md'; import SlSetUp from '/snippets/_new-sl-setup.md'; - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - ## Introduction The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), powered by [MetricFlow](/docs/build/about-metricflow), simplifies the setup of key business metrics. It centralizes definitions, avoids duplicate code, and ensures easy access to metrics in downstream tools. MetricFlow helps manage company metrics easier, allowing you to define metrics in your dbt project and query them in dbt Cloud with [MetricFlow commands](/docs/build/metricflow-commands). @@ -114,7 +106,7 @@ Open a new tab and follow these quick steps for account setup and data loading i -- Production and development environments must be on [dbt version 1.6 or higher](/docs/dbt-versions/upgrade-dbt-version-in-cloud). Alternatively, set your environment to "versionless" by selecting [ Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) to always get the latest updates. +- Production and development environments must be on [dbt version 1.6 or higher](/docs/dbt-versions/upgrade-dbt-version-in-cloud). Alternatively, set your environment to [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest updates. - Create a [trial Snowflake account](https://signup.snowflake.com/): - Select the Enterprise Snowflake edition with ACCOUNTADMIN access. Consider organizational questions when choosing a cloud provider, refer to Snowflake's [Introduction to Cloud Platforms](https://docs.snowflake.com/en/user-guide/intro-cloud-platforms). - Select a cloud provider and region. All cloud providers and regions will work so choose whichever you prefer. diff --git a/website/docs/guides/starburst-galaxy-qs.md b/website/docs/guides/starburst-galaxy-qs.md index e730abd1fed..3863d80a1b8 100644 --- a/website/docs/guides/starburst-galaxy-qs.md +++ b/website/docs/guides/starburst-galaxy-qs.md @@ -202,7 +202,7 @@ To query the Jaffle Shop data with Starburst Galaxy, you need to create tables u If this role is not listed for you, choose the role you selected in [Connect Starburst Galaxy to the Amazon S3 bucket](#connect-to-s3-bucket) when you added location privilege for your S3 bucket. 3. Click **Clusters** on the left sidebar. 4. Find your cluster in the **View clusters** table and click **Connection info**. Choose **dbt** from the **Select client** dropdown. Keep the **Connection information** modal open. You will use details from that modal in dbt Cloud. -5. In another browser tab, log in to [dbt Cloud](https://cloud.getdbt.com/). +5. In another browser tab, log in to [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). 6. Create a new project in dbt Cloud. From Account settings (using the gear menu in the top right corner), click **+ New Project**. 7. Enter a project name and click **Continue**. 8. Choose **Starburst** as your connection and click **Next**. diff --git a/website/docs/guides/zapier-ms-teams.md b/website/docs/guides/zapier-ms-teams.md index d841ca3305a..171ed19193a 100644 --- a/website/docs/guides/zapier-ms-teams.md +++ b/website/docs/guides/zapier-ms-teams.md @@ -105,7 +105,7 @@ run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Cloud Admin API -url = f'https://cloud.getdbt.com/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' +url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() diff --git a/website/docs/guides/zapier-slack.md b/website/docs/guides/zapier-slack.md index fccce59c948..b1016d4c969 100644 --- a/website/docs/guides/zapier-slack.md +++ b/website/docs/guides/zapier-slack.md @@ -100,7 +100,7 @@ run_id = hook_data['runId'] account_id = full_body['accountId'] # Fetch run info from the dbt Cloud Admin API -url = f'https://cloud.getdbt.com/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' +url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} run_data_response = requests.get(url, headers=headers) run_data_response.raise_for_status() @@ -260,7 +260,7 @@ api_token = secret_store.get('DBT_CLOUD_SERVICE_TOKEN') commands_to_skip_logs = ['dbt source', 'dbt docs'] run_id = input_data['run_id'] account_id = input_data['account_id'] -url = f'https://cloud.getdbt.com/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' +url = f'https://YOUR_ACCESS_URL/api/v2/accounts/{account_id}/runs/{run_id}/?include_related=["run_steps"]' headers = {'Authorization': f'Token {api_token}'} response = requests.get(url, headers=headers) diff --git a/website/docs/reference/artifacts/other-artifacts.md b/website/docs/reference/artifacts/other-artifacts.md index 75a4653d685..0216acccff0 100644 --- a/website/docs/reference/artifacts/other-artifacts.md +++ b/website/docs/reference/artifacts/other-artifacts.md @@ -25,14 +25,6 @@ Stores the network representation of the dbt resource DAG. ### graph_summary.json - - -:::info New functionality -This functionality is new in v1.6. -::: - - - **Produced by:** [manifest commands](/reference/artifacts/manifest-json) This file is useful for investigating performance issues in dbt Core's graph algorithms. diff --git a/website/docs/reference/artifacts/run-results-json.md b/website/docs/reference/artifacts/run-results-json.md index 75ff3412bfc..ff8da3559fa 100644 --- a/website/docs/reference/artifacts/run-results-json.md +++ b/website/docs/reference/artifacts/run-results-json.md @@ -12,7 +12,8 @@ sidebar_label: "Run results" [`run`](/reference/commands/run) [`seed`](/reference/commands/seed) [`snapshot`](/reference/commands/snapshot) - [`test`](/reference/commands/test) [`run-operation`](/reference/commands/run-operation) + [`test`](/reference/commands/test) + [`run-operation`](/reference/commands/run-operation) This file contains information about a completed invocation of dbt, including timing and status info for each node (model, test, etc) that was executed. In aggregate, many `run_results.json` can be combined to calculate average model runtime, test failure rates, the number of record changes captured by snapshots, etc. diff --git a/website/docs/reference/commands/cmd-docs.md b/website/docs/reference/commands/cmd-docs.md index 60b3049ccf2..cceb8c2ec6e 100644 --- a/website/docs/reference/commands/cmd-docs.md +++ b/website/docs/reference/commands/cmd-docs.md @@ -38,8 +38,6 @@ Use the `--no-compile` argument to skip re-compilation. When this flag is provid dbt docs generate --no-compile ``` - - Use the `--empty-catalog` argument to skip running the database queries to populate `catalog.json`. When this flag is provided, `dbt docs generate` will skip step (3) described above. This is not recommended for production environments, as it means that your documentation will be missing information gleaned from database metadata (the full set of columns in each table, and statistics about those tables). It can speed up `docs generate` in development, when you just want to visualize lineage and other information defined within your project. To learn how to build your documentation in dbt Cloud, refer to [build your docs in dbt Cloud](/docs/collaborate/build-and-view-your-docs). @@ -49,8 +47,6 @@ This is not recommended for production environments, as it means that your docum dbt docs generate --empty-catalog ``` - - ### dbt docs serve This command starts a webserver on port 8080 to serve your documentation locally and opens the documentation site in your default browser. The webserver is rooted in your `target/` directory. Be sure to run `dbt docs generate` before `dbt docs serve` because the `generate` command produces a [catalog metadata artifact](/reference/artifacts/catalog-json) that the `serve` command depends upon. You will see an error message if the catalog is missing. diff --git a/website/docs/reference/commands/compile.md b/website/docs/reference/commands/compile.md index 9422b64b1ea..d67ae80519a 100644 --- a/website/docs/reference/commands/compile.md +++ b/website/docs/reference/commands/compile.md @@ -16,8 +16,6 @@ Some common misconceptions: - `dbt compile` is _not_ a pre-requisite of `dbt run`, or other building commands. Those commands will handle compilation themselves. - If you just want dbt to read and validate your project code, without connecting to the data warehouse, use `dbt parse` instead. - - ### Interactive compile Starting in dbt v1.5, `compile` can be "interactive" in the CLI, by displaying the compiled code of a node or arbitrary dbt-SQL query: @@ -78,8 +76,6 @@ dbt compile --inline "select * from {{ ref('raw_orders') }}" select * from "jaffle_shop"."main"."raw_orders" ``` - - The command accesses the data platform to cache-related metadata, and to run introspective queries. Use the flags: - `--no-populate-cache` to disable the initial cache population. If metadata is needed, it will be a cache miss, requiring dbt to run the metadata query. This is a `dbt` flag, which means you need to add `dbt` as a prefix. For example: `dbt --no-populate-cache`. - `--no-introspect` to disable [introspective queries](/faqs/warehouse/db-connection-dbt-compile#introspective-queries). dbt will raise an error if a model's definition requires running one. This is a `dbt compile` flag, which means you need to add `dbt compile` as a prefix. For example:`dbt compile --no-introspect`. diff --git a/website/docs/reference/commands/debug.md b/website/docs/reference/commands/debug.md index 5f3ee3d5902..7317806f481 100644 --- a/website/docs/reference/commands/debug.md +++ b/website/docs/reference/commands/debug.md @@ -11,16 +11,12 @@ id: "debug" ## Example usage - - Only test the connection to the data platform and skip the other checks `dbt debug` looks for: ```shell $ dbt debug --connection ``` - - Show the configured location for the `profiles.yml` file and exit: ```text diff --git a/website/docs/reference/commands/list.md b/website/docs/reference/commands/list.md index e73699dc78c..9d77d4f348f 100644 --- a/website/docs/reference/commands/list.md +++ b/website/docs/reference/commands/list.md @@ -73,8 +73,6 @@ $ dbt ls --select snowplow.* --output json **Listing JSON output with custom keys** - - ``` $ dbt ls --select snowplow.* --output json --output-keys "name resource_type description" {"name": "snowplow_events", "description": "This is a pretty cool model", ...} @@ -82,10 +80,6 @@ $ dbt ls --select snowplow.* --output json --output-keys "name resource_type des ... ``` - - - - **Listing Semantic models** List all resources upstream of your orders semantic model: @@ -93,8 +87,6 @@ List all resources upstream of your orders semantic model: dbt ls -s +semantic_model:orders ``` - - **Listing file paths** ``` dbt ls --select snowplow.* --output path diff --git a/website/docs/reference/commands/parse.md b/website/docs/reference/commands/parse.md index e709e35aeb5..5e8145762f7 100644 --- a/website/docs/reference/commands/parse.md +++ b/website/docs/reference/commands/parse.md @@ -9,16 +9,12 @@ The `dbt parse` command parses and validates the contents of your dbt project. I It will also produce an artifact with detailed timing information, which is useful to understand parsing times for large projects. Refer to [Project parsing](/reference/parsing) for more information. - - Starting in v1.5, `dbt parse` will write or return a [manifest](/reference/artifacts/manifest-json), enabling you to introspect dbt's understanding of all the resources in your project. By default, the dbt Cloud IDE will attempt a "partial" parse, which means it'll only check changes since the last parse (new or updated parts of your project when you make changes). Since the dbt Cloud IDE automatically parses in the background whenever you save your work, manually running `dbt parse` yourself is likely to be fast because it's just looking at recent changes. As an option, you can tell dbt to check the entire project from scratch by using the `--no-partial-parse` flag. This makes dbt perform a full re-parse of the project, not just the recent changes. - - ``` $ dbt parse 13:02:52 Running with dbt=1.5.0 diff --git a/website/docs/reference/commands/version.md b/website/docs/reference/commands/version.md index 87a0f9ed975..794bb8e0284 100644 --- a/website/docs/reference/commands/version.md +++ b/website/docs/reference/commands/version.md @@ -13,7 +13,7 @@ The `--version` command-line flag returns information about the currently instal ## Versioning To learn more about release versioning for dbt Core, refer to [How dbt Core uses semantic versioning](docs/dbt-versions/core#how-dbt-core-uses-semantic-versioning). -If dbt Cloud is set to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version), then `dbt_version` uses the latest (continuous) release version. This also follows semantic versioning guidelines, using the `YYYY.xx.yy` format, where the year is the major version (for example, `2024.04.1234`) +If using [versionless dbt Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless), then `dbt_version` uses the latest (continuous) release version. This also follows semantic versioning guidelines, using the `YYYY.xx.yy` format, where the year is the major version (for example, `2024.04.1234`) ## Example usages diff --git a/website/docs/reference/data-test-configs.md b/website/docs/reference/data-test-configs.md index d98392a350b..577908ce031 100644 --- a/website/docs/reference/data-test-configs.md +++ b/website/docs/reference/data-test-configs.md @@ -278,7 +278,11 @@ tests: #### Specify custom configurations for generic data tests -_Currently available in dbt Cloud only. Specifying custom configurations for data tests will become available in dbt Core later this year._ +:::note + +This functionality is supported on ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless). Specifying custom configurations for data tests will become available in dbt Core v1.9, available later this year. + +::: Use any custom config key to specify custom configurations for data tests. For example, the following specifies the `snowflake_warehouse` custom config that dbt should use when executing the `accepted_values` data test: @@ -297,4 +301,4 @@ models: ``` -Given the config, the data test runs on a different Snowflake virtual warehouse than the one in your default connection to enable better price-performance with a different warehouse size or more granular cost allocation and visibility. \ No newline at end of file +Given the config, the data test runs on a different Snowflake virtual warehouse than the one in your default connection to enable better price-performance with a different warehouse size or more granular cost allocation and visibility. diff --git a/website/docs/reference/dbt-commands.md b/website/docs/reference/dbt-commands.md index 719c23d65d6..8386cf61731 100644 --- a/website/docs/reference/dbt-commands.md +++ b/website/docs/reference/dbt-commands.md @@ -26,8 +26,6 @@ dbt commands can be `read` or `write` commands: ## Available commands - - The following sections outline the commands supported by dbt and their relevant flags. They are available in all tools and all [supported versions](/docs/dbt-versions/core) unless noted otherwise. You can run these commands in your specific tool by prefixing them with `dbt` — for example, to run the `test` command, type `dbt test`. For information about selecting models on the command line, refer to [Model selection syntax](/reference/node-selection/syntax). @@ -59,58 +57,3 @@ Commands with a ('❌') indicate write commands, commands with a ('✅') indicat | [source](/reference/commands/source) | Provides tools for working with source data (including validating that sources are "fresh") | ✅ | All tools
All [supported versions](/docs/dbt-versions/core) | | [test](/reference/commands/test) | Executes tests defined in a project | ✅ | All tools
All [supported versions](/docs/dbt-versions/core) | Note, use the [`--version`](/reference/commands/version) flag to display the installed dbt Core or dbt Cloud CLI version. (Not applicable for the dbt Cloud IDE). Available on all [supported versions](/docs/dbt-versions/core). -
- - - -Select the tabs that are relevant to your development workflow. For example, if you develop in the dbt Cloud IDE, select **dbt Cloud**. - - - - -Use the following dbt commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`. - -- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests) -- [clone](/reference/commands/clone): clone selected nodes from the specified state (requires dbt 1.6 or higher) -- [compile](/reference/commands/compile): compiles (but does not run) the models in a project -- [deps](/reference/commands/deps): downloads dependencies for a project -- [docs](/reference/commands/cmd-docs) : generates documentation for a project -- [retry](/reference/commands/retry): retry the last run `dbt` command from the point of failure (requires dbt 1.6 or later) -- [run](/reference/commands/run): runs the models in a project -- [run-operation](/reference/commands/run-operation): invoke a macro, including running arbitrary maintenance SQL against the database -- [seed](/reference/commands/seed): loads CSV files into the database -- [show](/reference/commands/show): preview table rows post-transformation -- [snapshot](/reference/commands/snapshot): executes "snapshot" jobs defined in a project -- [source](/reference/commands/source): provides tools for working with source data (including validating that sources are "fresh") -- [test](/reference/commands/test): executes tests defined in a project - - - - - -Use the following dbt commands in [dbt Core](/docs/core/installation-overview) and use the `dbt` prefix. For example, to run the `test` command, type `dbt test`. - -- [build](/reference/commands/build): build and test all selected resources (models, seeds, snapshots, tests) -- [clean](/reference/commands/clean): deletes artifacts present in the dbt project -- [clone](/reference/commands/clone): clone selected models from the specified state (requires dbt 1.6 or higher) -- [compile](/reference/commands/compile): compiles (but does not run) the models in a project -- [debug](/reference/commands/debug): debugs dbt connections and projects -- [deps](/reference/commands/deps): downloads dependencies for a project -- [docs](/reference/commands/cmd-docs) : generates documentation for a project -- [init](/reference/commands/init): initializes a new dbt project -- [list](/reference/commands/list): lists resources defined in a dbt project -- [parse](/reference/commands/parse): parses a project and writes detailed timing info -- [retry](/reference/commands/retry): retry the last run `dbt` command from the point of failure (requires dbt 1.6 or higher) -- [rpc](/reference/commands/rpc): runs an RPC server that clients can submit queries to -- [run](/reference/commands/run): runs the models in a project -- [run-operation](/reference/commands/run-operation): invoke a macro, including running arbitrary maintenance SQL against the database -- [seed](/reference/commands/seed): loads CSV files into the database -- [show](/reference/commands/show): preview table rows post-transformation -- [snapshot](/reference/commands/snapshot): executes "snapshot" jobs defined in a project -- [source](/reference/commands/source): provides tools for working with source data (including validating that sources are "fresh") -- [test](/reference/commands/test): executes tests defined in a project - - - - - diff --git a/website/docs/reference/dbt-jinja-functions/builtins.md b/website/docs/reference/dbt-jinja-functions/builtins.md index d3241ab849a..fa5f57a48e0 100644 --- a/website/docs/reference/dbt-jinja-functions/builtins.md +++ b/website/docs/reference/dbt-jinja-functions/builtins.md @@ -21,8 +21,6 @@ The `builtins` variable is a dictionary containing the following keys: Using the `builtins` variable in this way is an advanced development workflow. Users should be ready to maintain and update these overrides when upgrading in the future. ::: - - From dbt v1.5 and higher, use the following macro to override the `ref` method available in the model compilation context to return a [Relation](/reference/dbt-classes#relation) with the database name overriden to `dev`. It includes logic to extract user-provided arguments, including version, and call the builtins.ref() function with either a single modelname argument or both packagename and modelname arguments, based on the number of positional arguments in varargs. @@ -59,7 +57,6 @@ Note that the `ref`, `source`, and `config` functions can't be overridden with a {% endmacro %} ``` - Logic within the ref macro can also be used to control which elements of the model path are rendered when run, for example the following logic renders only the schema and object identifier, but not the database reference i.e. `my_schema.my_model` rather than `my_database.my_schema.my_model`. This is especially useful when using snowflake as a warehouse, if you intend to change the name of the database post-build and wish the references to remain accurate. diff --git a/website/docs/reference/dbt-jinja-functions/env_var.md b/website/docs/reference/dbt-jinja-functions/env_var.md index a239ce4ff13..28feccc30e4 100644 --- a/website/docs/reference/dbt-jinja-functions/env_var.md +++ b/website/docs/reference/dbt-jinja-functions/env_var.md @@ -58,7 +58,7 @@ models: ### Secrets -For certain configurations, you can use "secret" env vars. Any env var named with the prefix `DBT_ENV_SECRET_``DBT_ENV_SECRET` will be: +For certain configurations, you can use "secret" env vars. Any env var named with the prefix `DBT_ENV_SECRET` will be: - Available for use in `profiles.yml` + `packages.yml`, via the same `env_var()` function - Disallowed everywhere else, including `dbt_project.yml` and model SQL, to prevent accidentally writing these secret values to the or metadata artifacts - Scrubbed from dbt logs and replaced with `*****`, any time its value appears in those logs (even if the env var was not called directly) @@ -98,5 +98,5 @@ select 1 as id ### dbt Cloud usage -If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET`). Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. +If you are using dbt Cloud, you must adhere to the naming conventions for environment variables. Environment variables in dbt Cloud must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`). Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. diff --git a/website/docs/reference/dbt-jinja-functions/flags.md b/website/docs/reference/dbt-jinja-functions/flags.md index 534a0fa8987..9df6f9a58ea 100644 --- a/website/docs/reference/dbt-jinja-functions/flags.md +++ b/website/docs/reference/dbt-jinja-functions/flags.md @@ -48,28 +48,6 @@ select 1 as id - - -```shell -$ DBT_ENV_CUSTOM_ENV_MYVAR=myvalue dbt compile -s my_model -``` - - - -```sql --- invocation_args_dict: --- {'write_json': True, 'use_colors': True, 'printer_width': 80, 'version_check': True, 'partial_parse': True, 'static_parser': True, 'profiles_dir': '/Users/.../.dbt', 'send_anonymous_usage_stats': False, 'event_buffer_size': 100000, 'quiet': False, 'no_print': False, 'parse_only': False, 'which': 'compile', 'rpc_method': 'compile', 'indirect_selection': 'eager'} - --- dbt_metadata_envs: --- {'MYVAR': 'myvalue'} - -select 1 as id -``` - - - - - The `invocation_command` key within `invocation_args_dict` includes the entire subcommand when it compiles: ```shell @@ -89,8 +67,4 @@ $ DBT_ENV_CUSTOM_ENV_MYVAR=myvalue dbt compile -s my_model -- {'MYVAR': 'myvalue'} select 1 as id -``` - - - - +``` \ No newline at end of file diff --git a/website/docs/reference/dbt-jinja-functions/graph.md b/website/docs/reference/dbt-jinja-functions/graph.md index bea09c326e8..ccb05da8f2b 100644 --- a/website/docs/reference/dbt-jinja-functions/graph.md +++ b/website/docs/reference/dbt-jinja-functions/graph.md @@ -23,8 +23,6 @@ to understand how to effectively use this variable. The `graph` context variable is a dictionary which maps node ids onto dictionary representations of those nodes. A simplified example might look like: - - ```json { "nodes": { @@ -81,8 +79,6 @@ representations of those nodes. A simplified example might look like: } ``` - - The exact contract for these model and source nodes is not currently documented, but that will change in the future. diff --git a/website/docs/reference/dbt-jinja-functions/model.md b/website/docs/reference/dbt-jinja-functions/model.md index 59c3421d856..903851617f2 100644 --- a/website/docs/reference/dbt-jinja-functions/model.md +++ b/website/docs/reference/dbt-jinja-functions/model.md @@ -11,7 +11,7 @@ description: "`model` is the dbt graph object (or node) for the current model." For example: ```jinja -{% if model.config.materialization = 'view' %} +{% if model.config.materialization == 'view' %} {{ log(model.name ~ " is a view.", info=True) }} {% endif %} ``` diff --git a/website/docs/reference/dbt-jinja-functions/ref.md b/website/docs/reference/dbt-jinja-functions/ref.md index 8212359c1f1..535093ead27 100644 --- a/website/docs/reference/dbt-jinja-functions/ref.md +++ b/website/docs/reference/dbt-jinja-functions/ref.md @@ -94,12 +94,8 @@ select * from {{ ref('project_or_package', 'model_name') }} We recommend using two-argument `ref` any time you are referencing a model defined in a different package or project. While not required in all cases, it's more explicit for you, for dbt, and future readers of your code. - - We especially recommend using two-argument `ref` to avoid ambiguity, in cases where a model name is duplicated across multiple projects or installed packages. If you use one-argument `ref` (just the `model_name`), dbt will look for a model by that name in the same namespace (package or project); if it finds none, it will raise an error. - - **Note:** The `project_or_package` should match the `name` of the project/package, as defined in its `dbt_project.yml`. This might be different from the name of the repository. It never includes the repository's organization name. For example, if you use the [`fivetran/stripe`](https://hub.getdbt.com/fivetran/stripe/latest/) package, the package name is `stripe`, not `fivetran/stripe`. ### Forcing Dependencies diff --git a/website/docs/reference/dbt-jinja-functions/run_query.md b/website/docs/reference/dbt-jinja-functions/run_query.md index 87970e024ed..a53927db13d 100644 --- a/website/docs/reference/dbt-jinja-functions/run_query.md +++ b/website/docs/reference/dbt-jinja-functions/run_query.md @@ -8,7 +8,7 @@ description: "Use `run_query` macro to run queries and fetch results." The `run_query` macro provides a convenient way to run queries and fetch their results. It is a wrapper around the [statement block](/reference/dbt-jinja-functions/statement-blocks), which is more flexible, but also more complicated to use. __Args__: - * `sql`: The SQL query to execute + * `query`: The SQL query to execute Returns a [Table](https://agate.readthedocs.io/page/api/table.html) object with the result of the query. If the specified query does not return results (eg. a , , or maintenance query), then the return value will be `none`. diff --git a/website/docs/reference/dbt_project.yml.md b/website/docs/reference/dbt_project.yml.md index 6166be1df6d..08261dd6932 100644 --- a/website/docs/reference/dbt_project.yml.md +++ b/website/docs/reference/dbt_project.yml.md @@ -3,13 +3,10 @@ Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — thi - dbt uses [YAML](https://yaml.org/) in a few different places. If you're new to YAML, it would be worth learning how arrays, dictionaries, and strings are represented. - - - By default, dbt looks for the `dbt_project.yml` in your current working directory and its parents, but you can set a different directory using the `--project-dir` flag or the `DBT_PROJECT_DIR` environment variable. -- Specify your dbt Cloud project ID in the `dbt_project.yml` file using `project-id` under the `dbt-cloud` config. Find your project ID in your dbt Cloud project URL: For example, in `https://cloud.getdbt.com/11/projects/123456`, the project ID is `123456`. +- Specify your dbt Cloud project ID in the `dbt_project.yml` file using `project-id` under the `dbt-cloud` config. Find your project ID in your dbt Cloud project URL: For example, in `https://YOUR_ACCESS_URL/11/projects/123456`, the project ID is `123456`. - - Note, you can't set up a "property" in the `dbt_project.yml` file if it's not a config (an example is [macros](/reference/macro-properties)). This applies to all types of resources. Refer to [Configs and properties](/reference/configs-and-properties) for more detail. @@ -99,7 +96,7 @@ vars:
- + @@ -169,76 +166,6 @@ vars: - - - - - -```yml -[name](/reference/project-configs/name): string - -[config-version](/reference/project-configs/config-version): 2 -[version](/reference/project-configs/version): version - -[profile](/reference/project-configs/profile): profilename - -[model-paths](/reference/project-configs/model-paths): [directorypath] -[seed-paths](/reference/project-configs/seed-paths): [directorypath] -[test-paths](/reference/project-configs/test-paths): [directorypath] -[analysis-paths](/reference/project-configs/analysis-paths): [directorypath] -[macro-paths](/reference/project-configs/macro-paths): [directorypath] -[snapshot-paths](/reference/project-configs/snapshot-paths): [directorypath] -[docs-paths](/reference/project-configs/docs-paths): [directorypath] -[asset-paths](/reference/project-configs/asset-paths): [directorypath] - -[target-path](/reference/global-configs/json-artifacts): directorypath -[log-path](/reference/global-configs/logs): directorypath -[packages-install-path](/reference/project-configs/packages-install-path): directorypath - -[clean-targets](/reference/project-configs/clean-targets): [directorypath] - -[query-comment](/reference/project-configs/query-comment): string - -[require-dbt-version](/reference/project-configs/require-dbt-version): version-range | [version-range] - -[quoting](/reference/project-configs/quoting): - database: true | false - schema: true | false - identifier: true | false - -models: - [](/reference/model-configs) - -seeds: - [](/reference/seed-configs) - -snapshots: - [](/reference/snapshot-configs) - -sources: - [](source-configs) - -tests: - [](/reference/data-test-configs) - -vars: - [](/docs/build/project-variables) - -[on-run-start](/reference/project-configs/on-run-start-on-run-end): sql-statement | [sql-statement] -[on-run-end](/reference/project-configs/on-run-start-on-run-end): sql-statement | [sql-statement] - -[dispatch](/reference/project-configs/dispatch-config): - - macro_namespace: packagename - search_order: [packagename] - -[restrict-access](/docs/collaborate/govern/model-access): true | false - -``` - - - - - ## Naming convention It's important to follow the correct YAML naming conventions for the configs in your `dbt_project.yml` file to ensure dbt can process them properly. This is especially true for resource types with more than one word. diff --git a/website/docs/reference/events-logging.md b/website/docs/reference/events-logging.md index de79d8d9171..f78774e6310 100644 --- a/website/docs/reference/events-logging.md +++ b/website/docs/reference/events-logging.md @@ -65,7 +65,7 @@ Many events are fired while compiling or running a specific DAG node (model, see | `node_finished_at` | Timestamp when node processing completed | | `node_name` | Name of this model/seed/test/etc | | `node_path` | File path to where this resource is defined | -| `node_relation` | Nested object containing this node's database representation: `database`, `schema`, `alias`, and full `relation_name` with quoting & inclusion policies applied | +| `node_relation` | Nested object containing this node's database representation: `database`, `schema`, `alias`, and full `relation_name` with quoting & inclusion policies applied | | `node_started_at` | Timestamp when node processing started | | `node_status` | Current status of the node, either `RunningStatus` (while running) or `NodeStatus` (finished) as defined in [the result contract](https://github.com/dbt-labs/dbt-core/blob/eba90863ed4043957330ea44ca267db1a2d81fcd/core/dbt/contracts/results.py#L75-L88) | | `resource_type` | `model`, `test`, `seed`, `snapshot`, etc. | @@ -121,10 +121,7 @@ Many events are fired while compiling or running a specific DAG node (model, see Older versions of `dbt-core` made available a full history of events fired during an invocation, in the form of an `EVENT_HISTORY` object. - - When [invoking dbt programmatically](programmatic-invocations#registering-callbacks), it is possible to register a callback on dbt's `EventManager`. This allows access to structured events as Python objects, to enable custom logging and integration with other systems. - The Python interface into events is significantly less mature than the structured logging interface. For all standard use cases, we recommend parsing JSON-formatted logs. diff --git a/website/docs/reference/global-configs/cache.md b/website/docs/reference/global-configs/cache.md index 7687df30339..1a74fef8d30 100644 --- a/website/docs/reference/global-configs/cache.md +++ b/website/docs/reference/global-configs/cache.md @@ -4,8 +4,6 @@ id: "cache" sidebar: "Cache" --- - - ### Cache population At the start of runs, dbt caches metadata about all the objects in all the schemas where it might materialize resources (such as models). By default, dbt populates the cache with information on all schemas related to the project. @@ -28,6 +26,3 @@ Or, to improve speed and performance while focused on developing Salesforce mode dbt --cache-selected-only run --select salesforce ``` - - - diff --git a/website/docs/reference/global-configs/indirect-selection.md b/website/docs/reference/global-configs/indirect-selection.md index 07116d4c4b1..729176a1ff4 100644 --- a/website/docs/reference/global-configs/indirect-selection.md +++ b/website/docs/reference/global-configs/indirect-selection.md @@ -16,8 +16,6 @@ When all flags are set, the order of precedence is as follows. Refer to [About g You can set the flag to: `empty`, `buildable`, `cautious`, or `eager` (default). By default, dbt indirectly selects all tests if they touch any resource you select. Learn more about these options in [Indirect selection in Test selection examples](/reference/node-selection/test-selection-examples?indirect-selection-mode=eager#indirect-selection). - - The following is a visualization of the impact `--indirect-selection` and the various flags have using three models, three tests, and `dbt build` as an example: @@ -36,8 +34,6 @@ The following is a visualization of the impact `--indirect-selection` and the va - - For example, you can run tests that only refer to selected nodes using a CLI configuration: diff --git a/website/docs/reference/global-configs/legacy-behaviors.md b/website/docs/reference/global-configs/legacy-behaviors.md index 56dc58f074c..1450fda1459 100644 --- a/website/docs/reference/global-configs/legacy-behaviors.md +++ b/website/docs/reference/global-configs/legacy-behaviors.md @@ -33,7 +33,7 @@ flags: -When we use dbt Cloud in the following table, we're referring to accounts that have gone versionless by opting to "[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)." +When we use dbt Cloud in the following table, we're referring to accounts that have gone "[Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless)." | Flag | dbt Cloud: Intro | dbt Cloud: Maturity | dbt Core: Intro | dbt Core: Maturity | |-----------------------------------------------------------------|------------------|---------------------|-----------------|--------------------| diff --git a/website/docs/reference/global-configs/logs.md b/website/docs/reference/global-configs/logs.md index 19ca8da6b5c..beca0ded49e 100644 --- a/website/docs/reference/global-configs/logs.md +++ b/website/docs/reference/global-configs/logs.md @@ -8,12 +8,8 @@ sidebar: "logs" dbt outputs logs to two different locations: CLI console and the log file. - - The `LOG_FORMAT` and `LOG_FORMAT_FILE` configs specify how dbt's logs should be formatted, and they each have the same options: `json`, `text`, and `debug`. - - ```text @@ -44,8 +40,6 @@ The `json` format outputs fully structured logs in the format {"data": {"adapter_name": "postgres", "adapter_version": "=1.8.0"}, "info": {"category": "", "code": "E034", "extra": {}, "invocation_id": "82131fa0-d2b4-4a77-9436-019834e22746", "level": "info", "msg": "Registered adapter: postgres=1.8.0", "name": "AdapterRegistered", "pid": 7875, "thread": "MainThread", "ts": "2024-05-29T23:32:56.437986Z"}} ``` - - When the `LOG_FORMAT` is set explicitly, it will take affect in both the console and log files whereas the `LOG_FORMAT_FILE` only affects the log file. @@ -56,8 +50,6 @@ dbt --log-format-file json run - - :::tip Tip: verbose structured logs Use `json` formatting value in conjunction with the `DEBUG` config to produce rich log information which can be piped into monitoring tools for analysis: @@ -70,8 +62,6 @@ See [structured logging](/reference/events-logging#structured-logging) for more ::: - - ### Log Level The `LOG_LEVEL` config sets the minimum severity of events captured in the console and file logs. This is a more flexible alternative to the `--debug` flag. The available options for the log levels are `debug`, `info`, `warn`, `error`, or `none`. @@ -90,9 +80,6 @@ To set the file log level as a different value than the console, use the `--log- dbt --log-level-file error run ``` - - - ### Debug-level logging The `DEBUG` config redirects dbt's debug logs to standard output. This has the effect of showing debug-level log information in the terminal in addition to the `logs/dbt.log` file. This output is verbose. @@ -156,8 +143,6 @@ The `LOG_CACHE_EVENTS` config allows detailed logging for [relational cache](ref dbt --log-cache-events compile ``` - - ### Color You can set the color preferences for the file logs only within `profiles.yml` or using the `--use-colors-file / --no-use-colors-file` flags. @@ -175,5 +160,3 @@ config: dbt --use-colors-file run dbt --no-use-colors-file run ``` - - diff --git a/website/docs/reference/global-configs/print-output.md b/website/docs/reference/global-configs/print-output.md index fc129b162a7..7675dbb82c2 100644 --- a/website/docs/reference/global-configs/print-output.md +++ b/website/docs/reference/global-configs/print-output.md @@ -6,8 +6,6 @@ sidebar: "Print output" ### Suppress `print()` messages in stdout - - By default, dbt includes [`print()`](/reference/dbt-jinja-functions/print) messages in standard out (stdout). You can use the `DBT_PRINT` environment variable to prevent these messages from showing up in stdout. :::warning Syntax deprecation @@ -16,8 +14,6 @@ The original `DBT_NO_PRINT` environment variable has been deprecated, starting w ::: - - Supply `--no-print` flag to `dbt run` to suppress `print()` messages from showing in stdout. ```text @@ -54,7 +50,6 @@ config: dbt --use-colors run dbt --no-use-colors run ``` - You can set the color preferences for the file logs only within `profiles.yml` or using the `--use-colors-file / --no-use-colors-file` flags. @@ -71,5 +66,3 @@ config: dbt --use-colors-file run dbt --no-use-colors-file run ``` - - diff --git a/website/docs/reference/global-configs/version-compatibility.md b/website/docs/reference/global-configs/version-compatibility.md index 756147745a1..80841678a85 100644 --- a/website/docs/reference/global-configs/version-compatibility.md +++ b/website/docs/reference/global-configs/version-compatibility.md @@ -14,7 +14,7 @@ Running with dbt=1.0.0 Found 13 models, 2 tests, 1 archives, 0 analyses, 204 macros, 2 operations.... ``` -:::info Keep on latest version +:::info Versionless ::: diff --git a/website/docs/reference/global-configs/warnings.md b/website/docs/reference/global-configs/warnings.md index 0cb4add5f0d..97eb270338e 100644 --- a/website/docs/reference/global-configs/warnings.md +++ b/website/docs/reference/global-configs/warnings.md @@ -82,6 +82,7 @@ DBT_WARN_ERROR_OPTIONS='{"include": ["NoNodesForSelectionCriteria"]}' dbt run ... ``` +Values for `error`, `warn`, and/or `silence` should be passed on as arrays. For example, `dbt --warn-error-options '{"error": "all", "warn": ["NoNodesForSelectionCriteria"]}' run` not `dbt --warn-error-options '{"error": "all", "warn": "NoNodesForSelectionCriteria"}' run`. diff --git a/website/docs/reference/node-selection/defer.md b/website/docs/reference/node-selection/defer.md index bbcc5f7d567..27cc3dd78c5 100644 --- a/website/docs/reference/node-selection/defer.md +++ b/website/docs/reference/node-selection/defer.md @@ -8,12 +8,8 @@ Defer requires that a manifest from a previous dbt invocation be passed to the ` An alternative command that accomplishes similar functionality for different use cases is `dbt clone` - see the docs for [clone](/reference/commands/clone#when-to-use-dbt-clone-instead-of-deferral) for more information. - - It is possible to use separate state for `state:modified` and `--defer`, by passing paths to different manifests to each of the `--state`/`DBT_STATE` and `--defer-state`/`DBT_DEFER_STATE`. This enables more granular control in cases where you want to compare against logical state from one environment or past point in time, and defer to applied state from a different environment or point in time. If `--defer-state` is not specified, deferral will use the manifest supplied to `--state`. In most cases, you will want to use the same state for both: compare logical changes against production, and also "fail over" to the production environment for unbuilt upstream resources. - - ### Usage ```shell @@ -43,11 +39,8 @@ When using defer, you may be selecting from production datasets, development dat - if you apply env-specific limits in dev but not prod, as you may end up selecting more data than you expect - when executing tests that depend on multiple parents (e.g. `relationships`), since you're testing "across" environments - - Deferral requires both `--defer` and `--state` to be set, either by passing flags explicitly or by setting environment variables (`DBT_DEFER` and `DBT_STATE`). If you use dbt Cloud, read about [how to set up CI jobs](/docs/deploy/continuous-integration). - #### Favor state diff --git a/website/docs/reference/node-selection/methods.md b/website/docs/reference/node-selection/methods.md index 438847c6d5b..37f50f734e7 100644 --- a/website/docs/reference/node-selection/methods.md +++ b/website/docs/reference/node-selection/methods.md @@ -6,8 +6,6 @@ Selector methods return all resources that share a common property, using the syntax `method:value`. While it is recommended to explicitly denote the method, you can omit it (the default value will be one of `path`, `file` or `fqn`). - - Many of the methods below support Unix-style wildcards: @@ -24,8 +22,6 @@ dbt list --select "*.folder_name.*" dbt list --select "package:*_source" ``` - - ### The "tag" method The `tag:` method is used to select models that match a specified [tag](/reference/resource-configs/tags). @@ -199,12 +195,8 @@ dbt test --select "test_name:range_min_max" # run all instances of a custom **N.B.** State-based selection is a powerful, complex feature. Read about [known caveats and limitations](/reference/node-selection/state-comparison-caveats) to state comparison. - - The `state` method is used to select nodes by comparing them against a previous version of the same project, which is represented by a [manifest](/reference/artifacts/manifest-json). The file path of the comparison manifest _must_ be specified via the `--state` flag or `DBT_STATE` environment variable. - - `state:new`: There is no node with the same `unique_id` in the comparison manifest `state:modified`: All new nodes, plus any changes to existing nodes. @@ -227,16 +219,12 @@ Because state comparison is complex, and everyone's project is different, dbt su Remember that `state:modified` includes _all_ of the criteria above, as well as some extra resource-specific criteria, such as modifying a source's `freshness` or `quoting` rules or an exposure's `maturity` property. (View the source code for the full set of checks used when comparing [sources](https://github.com/dbt-labs/dbt-core/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L660-L681), [exposures](https://github.com/dbt-labs/dbt-core/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L768-L783), and [executable nodes](https://github.com/dbt-labs/dbt-core/blob/9e796671dd55d4781284d36c035d1db19641cd80/core/dbt/contracts/graph/parsed.py#L319-L330).) - - There are two additional `state` selectors that complement `state:new` and `state:modified` by representing the inverse of those functions: - `state:old` — A node with the same `unique_id` exists in the comparison manifest - `state:unmodified` — All existing nodes with no changes These selectors can help you shorten run times by excluding unchanged nodes. Currently, no subselectors are available at this time, but that might change as use cases evolve. - - ### The "exposure" method The `exposure` method is used to select parent resources of a specified [exposure](/docs/build/exposures). Use in conjunction with the `+` operator. @@ -279,7 +267,6 @@ The following dbt commands produce `sources.json` artifacts whose results can be After issuing one of the above commands, you can reference the source freshness results by adding a selector to a subsequent command as follows: - ```bash # You can also set the DBT_STATE environment variable instead of the --state flag. @@ -287,13 +274,8 @@ dbt source freshness # must be run again to compare current to previous state dbt build --select "source_status:fresher+" --state path/to/prod/artifacts ``` - - - ### The "group" method - - The `group` method is used to select models defined within a [group](/reference/resource-configs/group). @@ -301,12 +283,8 @@ The `group` method is used to select models defined within a [group](/reference/ dbt run --select "group:finance" # run all models that belong to the finance group. ``` - - ### The "access" method - - The `access` method selects models based on their [access](/reference/resource-configs/access) property. ```bash @@ -315,12 +293,8 @@ dbt list --select "access:private" # list all private models dbt list --select "access:protected" # list all protected models ``` - - ### The "version" method - - The `version` method selects [versioned models](/docs/collaborate/govern/model-versions) based on their [version identifier](/reference/resource-properties/versions) and [latest version](/reference/resource-properties/latest_version). ```bash @@ -331,13 +305,7 @@ dbt list --select "version:old" # versions older than the 'latest' versi dbt list --select "version:none" # models that are *not* versioned ``` - - ### The "semantic_model" method - -Supported in v1.6 or newer. - - The `semantic_model` method selects [semantic models](/docs/build/semantic-models). @@ -346,8 +314,6 @@ dbt list --select "semantic_model:*" # list all semantic models dbt list --select "+semantic_model:orders" # list your semantic model named "orders" and all upstream resources ``` - - ### The "saved_query" method Supported in v1.7 or newer. diff --git a/website/docs/reference/node-selection/syntax.md b/website/docs/reference/node-selection/syntax.md index ce7d27cb6b6..a46c4145217 100644 --- a/website/docs/reference/node-selection/syntax.md +++ b/website/docs/reference/node-selection/syntax.md @@ -136,8 +136,6 @@ Together, the `state:` selector and deferral enable ["slim CI"](/best-practices/ State and defer can be set by environment variables as well as CLI flags: - - - `--state` or `DBT_STATE`: file path - `--defer` or `DBT_DEFER`: boolean @@ -147,18 +145,12 @@ In dbt v1.5, we deprecated the original syntax for state (`DBT_ARTIFACT_STATE_PA ::: - - - - - `--state` or `DBT_STATE`: file path - `--defer` or `DBT_DEFER`: boolean - `--defer-state` or `DBT_DEFER_STATE`: file path to use for deferral only (optional) If `--defer-state` is not specified, deferral will use the artifacts supplied by `--state`. This enables more granular control in cases where you want to compare against logical state from one environment or past point in time, and defer to applied state from a different environment or point in time. - - If both the flag and env var are provided, the flag takes precedence. #### Notes: diff --git a/website/docs/reference/node-selection/test-selection-examples.md b/website/docs/reference/node-selection/test-selection-examples.md index 11362b2364b..445da893ce5 100644 --- a/website/docs/reference/node-selection/test-selection-examples.md +++ b/website/docs/reference/node-selection/test-selection-examples.md @@ -35,19 +35,13 @@ In both cases, `test_type` checks a property of the test itself. These are forms ### Indirect selection - - - - ### Indirect selection examples - - To visualize these methods, suppose you have `model_a`, `model_b`, and `model_c` and associated data tests. The following illustrates which tests will be run when you execute `dbt build` with the various indirect selection modes: @@ -115,8 +109,6 @@ dbt build --select "orders" --indirect-selection=empty - - ### Test selection syntax examples diff --git a/website/docs/reference/node-selection/yaml-selectors.md b/website/docs/reference/node-selection/yaml-selectors.md index d911eb44baa..ff6628919b7 100644 --- a/website/docs/reference/node-selection/yaml-selectors.md +++ b/website/docs/reference/node-selection/yaml-selectors.md @@ -57,8 +57,6 @@ This is the most thorough syntax, which can include the operator-equivalent keyw Review [methods](/reference/node-selection/methods) for the available list. - - ```yml definition: method: tag @@ -77,9 +75,6 @@ definition: indirect_selection: eager | cautious | buildable | empty # include all tests selected indirectly? eager by default ``` - - - The `*` operator to select all nodes can be written as: ```yml definition: @@ -113,16 +108,11 @@ Note: The `exclude` argument in YAML selectors is subtly different from the `--exclude` CLI argument. Here, `exclude` _always_ returns a [set difference](https://en.wikipedia.org/wiki/Complement_(set_theory)), and it is always applied _last_ within its scope. - - When more than one "yeslist" (`--select`) is passed, they are treated as a [union](/reference/node-selection/set-operators#unions) rather than an [intersection](/reference/node-selection/set-operators#intersections). Same thing when there is more than one "nolist" (`--exclude`). - #### Indirect selection - - As a general rule, dbt will indirectly select _all_ tests if they touch _any_ resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious", "buildable", or "empty" by setting `indirect_selection` for a specific criterion: ```yml @@ -145,8 +135,6 @@ As a general rule, dbt will indirectly select _all_ tests if they touch _any_ re If provided, a YAML selector's `indirect_selection` value will take precedence over the CLI flag `--indirect-selection`. Because `indirect_selection` is defined separately for _each_ selection criterion, it's possible to mix eager/cautious/buildable/empty modes within the same definition, to achieve the exact behavior that you need. Remember that you can always test out your critiera with `dbt ls --selector`. - - See [test selection examples](/reference/node-selection/test-selection-examples) for more details about indirect selection. ## Example diff --git a/website/docs/reference/project-configs/config-version.md b/website/docs/reference/project-configs/config-version.md index c785b2d4c22..78b9be49a20 100644 --- a/website/docs/reference/project-configs/config-version.md +++ b/website/docs/reference/project-configs/config-version.md @@ -3,11 +3,7 @@ datatype: integer description: "Read this guide to understand the config-version configuration in dbt." --- - - -Starting in dbt v1.5, the `config-version:` tag is optional. - - +The `config-version:` tag is optional. @@ -18,9 +14,9 @@ config-version: 2 ## Definition -Specify your `dbt_project.yml` as using the v2 structure. - This configuration is optional. +Specify your `dbt_project.yml` as using the v2 structure. ## Default + Without this configuration, dbt will assume your `dbt_project.yml` uses the version 2 syntax. Version 1 was deprecated in dbt v0.19.0. diff --git a/website/docs/reference/project-configs/require-dbt-version.md b/website/docs/reference/project-configs/require-dbt-version.md index 5fb08927f0f..42dc49c4546 100644 --- a/website/docs/reference/project-configs/require-dbt-version.md +++ b/website/docs/reference/project-configs/require-dbt-version.md @@ -22,7 +22,7 @@ When you set this configuration, dbt sends a helpful error message for any user If this configuration is not specified, no version check will occur. -:::info Keep on latest version +:::info Versionless diff --git a/website/docs/reference/resource-configs/alias.md b/website/docs/reference/resource-configs/alias.md index 6cb14371dfa..3f36bbd0d8f 100644 --- a/website/docs/reference/resource-configs/alias.md +++ b/website/docs/reference/resource-configs/alias.md @@ -116,5 +116,7 @@ The standard behavior of dbt is: * If a custom alias is _not_ specified, the identifier of the relation is the resource name (i.e. the filename). * If a custom alias is specified, the identifier of the relation is the `{{ alias }}` value. +**Note** With an [ephemeral model](/docs/build/materializations), dbt will always apply the prefix `__dbt__cte__` to the identifier. This means that if an alias is set on an ephemeral model, then its CTE identifier will be `__dbt__cte__{{ alias }}`, but if no alias is set then its identifier will be `__dbt__cte__{{ filename }}`. + To learn more about changing the way that dbt generates a relation's `identifier`, read [Using Aliases](/docs/build/custom-aliases). diff --git a/website/docs/reference/resource-configs/bigquery-configs.md b/website/docs/reference/resource-configs/bigquery-configs.md index d3a7b6ef44a..081da58dddb 100644 --- a/website/docs/reference/resource-configs/bigquery-configs.md +++ b/website/docs/reference/resource-configs/bigquery-configs.md @@ -336,7 +336,7 @@ dbt supports the specification of BigQuery labels for the tables and BigQuery key-value pair entries for labels larger than 63 characters are truncated. + BigQuery key-value pair entries for labels larger than 63 characters are truncated. **Configuring labels in a model file** diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index fc2b2c81828..68224802026 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -7,20 +7,7 @@ id: "databricks-configs" When materializing a model as `table`, you may include several optional configs that are specific to the dbt-databricks plugin, in addition to the standard [model configs](/reference/model-configs). - - -| Option | Description | Required? | Example | -|---------------------|------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|--------------------------| -| file_format | The file format to use when creating tables (`parquet`, `delta`, `hudi`, `csv`, `json`, `text`, `jdbc`, `orc`, `hive` or `libsvm`). | Optional | `delta` | -| location_root | The created table uses the specified directory to store its data. The table alias is appended to it. | Optional | `/mnt/root` | -| partition_by | Partition the created table by the specified columns. A directory is created for each partition. | Optional | `date_day` | -| clustered_by | Each partition in the created table will be split into a fixed number of buckets by the specified columns. | Optional | `country_code` | -| buckets | The number of buckets to create while clustering | Required if `clustered_by` is specified | `8` | -| tblproperties | [Tblproperties](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-tblproperties.html) to be set on the created table | Optional | `{'this.is.my.key': 12}` | - - - - + | Option | Description | Required? | Model Support | Example | @@ -621,7 +608,7 @@ snapshots: - + ## Materialized views and streaming tables Starting with version 1.6.0, the dbt-databricks adapter supports [materialized views](https://docs.databricks.com/en/sql/user/materialized-views.html) and [streaming tables](https://docs.databricks.com/en/sql/load-data-streaming-table.html), as alternatives to incremental tables that are powered by [Delta Live Tables](https://docs.databricks.com/en/delta-live-tables/index.html). diff --git a/website/docs/reference/resource-configs/firebolt-configs.md b/website/docs/reference/resource-configs/firebolt-configs.md index 59adee715ba..394823e33de 100644 --- a/website/docs/reference/resource-configs/firebolt-configs.md +++ b/website/docs/reference/resource-configs/firebolt-configs.md @@ -14,7 +14,7 @@ seeds: ``` -## Model Configuration for Fact Tables +## Model configuration for fact tables A dbt model can be created as a Firebolt fact and configured using the following syntax: @@ -58,7 +58,7 @@ models: table_type: fact primary_index: [ , ... ] indexes: - - type: aggregating | join + - type: aggregating key_column: [ , ... ] aggregation: [ , ... ] ... @@ -91,20 +91,20 @@ models: -#### Fact Table Configurations +#### Fact table configurations | Configuration | Description | |-------------------|-------------------------------------------------------------------------------------------| | `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a fact table. | -| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/working-with-tables.html#fact-and-dimension-tables) table. | +| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. | | `primary_index` | Sets the primary index for the fact table using the inputted list of column names from the model. Required for fact tables. | | `indexes` | A list of aggregating indexes to create on the fact table. | -| `type` | Specifies whether the index is an aggregating index or join index. Join indexes only apply to dimension tables, so for fact tables set to `aggregating`. | +| `type` | Specifies that the index is an [aggregating index](https://docs.firebolt.io/godocs/Guides/working-with-indexes/using-aggregating-indexes.html). Should be set to `aggregating`. | | `key_column` | Sets the grouping of the aggregating index using the inputted list of column names from the model. | | `aggregation` | Sets the aggregations on the aggregating index using the inputted list of SQL agg expressions. | -#### Example of a Fact Table With an Aggregating Index +#### Example of a fact table with an aggregating index ``` {{ config( @@ -122,7 +122,7 @@ models: ``` -## Model Configuration for Dimension Tables +## Model configuration for dimension tables A dbt model can be materialized as a Firebolt dimension table and configured using the following syntax: @@ -144,11 +144,7 @@ models: : +materialized: table +table_type: dimension - +indexes: - - type: join - join_column: - dimension_column: [ , ... ] - ... + ... ``` @@ -163,11 +159,7 @@ models: config: materialized: table table_type: dimension - indexes: - - type: join - join_column: - dimension_column: [ , ... ], - ... + ... ``` @@ -180,14 +172,7 @@ models: {{ config( materialized = "table", table_type = "dimension", - indexes = [ - { - type = "join", - join_column = "", - dimension_column: [ "", ... ] - }, - ... - ], + ... ) }} ``` @@ -195,39 +180,19 @@ models: +Dimension tables do not support aggregation indexes. -#### Dimension Table Configurations +#### Dimension table configurations | Configuration | Description | |--------------------|-------------------------------------------------------------------------------------------| | `materialized` | How the model will be materialized into Firebolt. Must be `table` to create a dimension table. | -| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/working-with-tables.html#fact-and-dimension-tables) table. | -| `indexes` | A list of join indexes to create on the dimension table. | -| `type` | Specifies whether the index is an aggregating index or join index. Aggregating indexes only apply to fact tables, so for dimension tables set to `join`. | -| `join_column` | Sets the join key of the join index using the inputted column name from the model. | -| `dimension_column` | Sets the columns to be loaded into memory on the join index using the inputted list of column names from the mode. | +| `table_type` | Whether the materialized table will be a [fact or dimension](https://docs.firebolt.io/godocs/Overview/working-with-tables/working-with-tables.html#fact-and-dimension-tables) table. | -#### Example of a Dimension Table With a Join Index +## How aggregating indexes are named -``` -{{ config( - materialized = "table", - table_type = "dimension", - indexes = [ - { - type: "join", - join_column: "order_id", - dimension_column: ["customer_id", "status"] - } - ] -) }} -``` - - -## How Aggregating Indexes and Join Indexes Are Named - -In dbt-firebolt, you do not provide names for aggregating indexes and join indexes; they are named programmatically. dbt will generate index names using the following convention: +In dbt-firebolt, you do not provide names for aggregating indexes; they are named programmatically. dbt will generate index names using the following convention: ``` _____ @@ -236,14 +201,14 @@ In dbt-firebolt, you do not provide names for aggregating indexes and join index For example, a join index could be named `my_users__id__join_1633504263` and an aggregating index could be named `my_orders__order_date__aggregating_1633504263`. -## Managing Ingestion via External Tables +## Managing ingestion via external tables `dbt-firebolt` supports dbt's [external tables feature](https://docs.getdbt.com/reference/resource-properties/external), which allows dbt to manage the table ingestion process from S3 into Firebolt. This is an optional feature but can be highly convenient depending on your use case. -More information on using external tables including properly configuring IAM can be found in the Firebolt [documentation](https://docs.firebolt.io/sql-reference/commands/ddl-commands#create-external-table). +More information on using external tables including properly configuring IAM can be found in the Firebolt [documentation](https://docs.firebolt.io/godocs/Guides/loading-data/working-with-external-tables.html). -#### Installation of External Tables Package +#### Installation of external tables package To install and use `dbt-external-tables` with Firebolt, you must: @@ -266,14 +231,14 @@ To install and use `dbt-external-tables` with Firebolt, you must: 3. Pull in the `packages.yml` dependencies by calling `dbt deps`. -#### Using External Tables +#### Using external tables To use external tables, you must define a table as `external` in your `dbt_project.yml` file. Every external table must contain the fields `url`, `type`, and `object_pattern`. Note that the Firebolt external table specification requires fewer fields than what is specified in the dbt documentation. In addition to specifying the columns, an external table may specify partitions. Partitions are not columns and they cannot have the same name as columns. To avoid YAML parsing errors, remember to encase string literals (such as the `url` and `object_pattern` values) in single quotation marks. -#### dbt_project.yml Syntax For an External Table +#### dbt_project.yml syntax for an external table ```yml sources: @@ -288,8 +253,8 @@ sources: object_pattern: '' type: '' credentials: - internal_role_arn: arn:aws:iam::id:/ - external_role_id: + aws_key_id: + aws_secret_key: object_pattern: '' compression: '' partitions: @@ -301,7 +266,10 @@ sources: data_type: ``` -#### Running External tables +`aws_key_id` and `aws_secret_key` are the credentails that allow Firebolt access to your S3 bucket. Learn +how to set them up by following this [guide](https://docs.firebolt.io/godocs/Guides/loading-data/creating-access-keys-aws.html). If your bucket is public these parameters are not necessary. + +#### Running external tables The `stage_external_sources` macro is inherited from the [dbt-external-tables package](https://github.com/dbt-labs/dbt-external-tables#syntax) and is the primary point of entry when using thes package. It has two operational modes: standard and "full refresh." @@ -311,11 +279,11 @@ $ dbt run-operation stage_external_sources # iterate through all source nodes, create or replace (no refresh command is required as data is fetched live from remote) $ dbt run-operation stage_external_sources --vars "ext_full_refresh: true" -``` +``` ## Incremental models -The [`incremental_strategy` configuration](/docs/build/incremental-strategy) controls how dbt builds incremental models. Firebolt currently supports the `append` configuration. You can specify `incremental_strategy` in `dbt_project.yml` or within a model file's `config()` block. The `append` configuration is the default. Specifying this configuration is optional. +The [`incremental_strategy` configuration](/docs/build/incremental-strategy) controls how dbt builds incremental models. Firebolt currently supports `append`, `insert_overwrite` and `delete+insert` configuration. You can specify `incremental_strategy` in `dbt_project.yml` or within a model file's `config()` block. The `append` configuration is the default. Specifying this configuration is optional. The `append` strategy performs an `INSERT INTO` statement with all the new data based on the model definition. This strategy doesn't update or delete existing rows, so if you do not filter the data to the most recent records only, it is likely that duplicate records will be inserted. diff --git a/website/docs/reference/resource-configs/group.md b/website/docs/reference/resource-configs/group.md index 5fc5382760b..717d7de89f5 100644 --- a/website/docs/reference/resource-configs/group.md +++ b/website/docs/reference/resource-configs/group.md @@ -18,8 +18,6 @@ id: "group" }> - - ```yml @@ -60,14 +58,10 @@ select ... - - - - ```yml @@ -88,15 +82,10 @@ seeds: - - - - - ```yml @@ -123,15 +112,10 @@ select ... - - - - - ```yml @@ -184,14 +168,10 @@ select ... - - - - ```yml @@ -204,15 +184,11 @@ analyses: - - - - ```yaml @@ -237,8 +213,6 @@ metrics: - - diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index b7b88c525dd..07cfc938f1c 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -8,22 +8,10 @@ id: "postgres-configs" In dbt-postgres, the following incremental materialization strategies are supported: - - -- `append` (default when `unique_key` is not defined) -- `delete+insert` (default when `unique_key` is defined) - - - - - - `append` (default when `unique_key` is not defined) - `merge` - `delete+insert` (default when `unique_key` is defined) - - - ## Performance optimizations ### Unlogged @@ -104,8 +92,6 @@ models: - - ## Materialized views The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) @@ -200,7 +186,7 @@ This happens via a `DROP/CREATE` of the indexes, which can be thought of as an ` Learn more about these parameters in Postgres's [docs](https://www.postgresql.org/docs/current/sql-creatematerializedview.html). - + ### Limitations @@ -216,5 +202,3 @@ If the user changes the model's config to `materialized="materialized_view"`, th The solution is to execute `DROP TABLE my_model` on the data warehouse before trying the model again. - - diff --git a/website/docs/reference/resource-configs/redshift-configs.md b/website/docs/reference/resource-configs/redshift-configs.md index 78b288083fa..dcd87118d13 100644 --- a/website/docs/reference/resource-configs/redshift-configs.md +++ b/website/docs/reference/resource-configs/redshift-configs.md @@ -14,21 +14,10 @@ To-do: In dbt-redshift, the following incremental materialization strategies are supported: - - -- `append` (default when `unique_key` is not defined) -- `delete+insert` (default when `unique_key` is defined) - - - - - - `append` (default when `unique_key` is not defined) - `merge` - `delete+insert` (default when `unique_key` is defined) - - All of these strategies are inherited from dbt-postgres. ## Performance optimizations @@ -107,8 +96,6 @@ models: - - ## Materialized views The Redshift adapter supports [materialized views](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) @@ -241,7 +228,7 @@ As with most data platforms, there are limitations associated with materialized Find more information about materialized view limitations in Redshift's [docs](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-limitations). - + #### Changing materialization from "materialized_view" to "table" or "view" @@ -256,8 +243,6 @@ The workaround is to execute `DROP MATERIALIZED VIEW my_mv CASCADE` on the data - - ## Unit test limitations diff --git a/website/docs/reference/resource-configs/schema.md b/website/docs/reference/resource-configs/schema.md index 311dc01e767..57a357767cb 100644 --- a/website/docs/reference/resource-configs/schema.md +++ b/website/docs/reference/resource-configs/schema.md @@ -41,7 +41,8 @@ seeds: +schema: mappings ``` -This would result in the generated relation being located in the `mappings` schema, so the full relation name would be `analytics.target_schema_mappings.product_mappings`. +This would result in the generated relation being located in the `mappings` schema, so the full relation name would be `analytics.mappings.seed_name`. + @@ -149,14 +150,16 @@ seeds: ### Tests -Customize the name of the schema in which tests [configured to store failures](/reference/resource-configs/store_failures) will save their results: +Customize the name of the schema in which tests [configured to store failures](/reference/resource-configs/store_failures) will save their results. +The resulting schema is `{{ profile.schema }}_{{ tests.schema }}`, with a default suffix of `dbt_test__audit`. +To use the same profile schema, set `+schema: null`. ```yml tests: +store_failures: true - +schema: the_island_of_misfit_tests + +schema: _sad_test_failures # Will write tables to my_database.my_schema__sad_test_failures ``` diff --git a/website/docs/reference/resource-configs/singlestore-configs.md b/website/docs/reference/resource-configs/singlestore-configs.md index 2317c05c0e2..c1a1c6869f1 100644 --- a/website/docs/reference/resource-configs/singlestore-configs.md +++ b/website/docs/reference/resource-configs/singlestore-configs.md @@ -107,8 +107,6 @@ select ... - - ## Model contracts Starting from 1.5, the `dbt-singlestore` adapter supports model contracts. @@ -217,5 +215,3 @@ It's important to note that certain data type mappings might show up differently Just keep these points in mind when setting up and using your `dbt-singlestore` adapter, and you'll avoid common pitfalls! - - diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 4dc3fbcafef..a59bc8dee00 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -338,8 +338,6 @@ In the configuration format for the model SQL file: - - ## Dynamic tables The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about). @@ -574,7 +572,11 @@ As with materialized views on most data platforms, there are limitations associa Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions). - +For dbt limitations, these dbt features are not supported: +- [Model contracts](/docs/collaborate/govern/model-contracts) +- [Copy grants configuration](/reference/resource-configs/snowflake-configs#copying-grants) + + #### Changing materialization to and from "dynamic_table" @@ -601,9 +603,6 @@ The workaround is to execute `DROP TABLE my_model` on the data warehouse before - - - ## Source freshness known limitation Snowflake calculates source freshness using information from the `LAST_ALTERED` column, meaning it relies on a field updated whenever any object undergoes modification, not only data updates. No action must be taken, but analytics teams should note this caveat. diff --git a/website/docs/reference/resource-configs/store_failures.md b/website/docs/reference/resource-configs/store_failures.md index 633895b4fe2..26b5b372701 100644 --- a/website/docs/reference/resource-configs/store_failures.md +++ b/website/docs/reference/resource-configs/store_failures.md @@ -10,7 +10,7 @@ Optionally set a test to always or never store its failures in the database. - If specified as `true` or `false`, the `store_failures` config will take precedence over the presence or absence of the `--store-failures` flag. - If the `store_failures` config is `none` or omitted, the resource will use the value of the `--store-failures` flag. -- When true, `store_failures` saves all the record(s) that failed the test only if [limit](/reference/resource-configs/limit) is not set or if there are fewer records than the limit. `store_failures` are saved in a new table with the name of the test. +- When true, `store_failures` saves all records (up to [limit](/reference/resource-configs/limit)) that failed the test. Failures are saved in a new table with the name of the test. By default, `store_failures` uses the schema `{{ profile.schema }}_dbt_test__audit`, but you can [configure](/reference/resource-configs/schema#tests) the schema suffix to a different value. - A test's results will always **replace** previous failures for the same test, even if that test results in no failures. - By default, `store_failures` uses a schema named `dbt_test__audit`, but, you can [configure](/reference/resource-configs/schema#tests) the schema to a different value. Ensure you have the authorization to create or access schemas for your work. For more details, refer to the [FAQ](#faqs). diff --git a/website/docs/reference/resource-configs/target_database.md b/website/docs/reference/resource-configs/target_database.md index 773d7815743..3c07b442107 100644 --- a/website/docs/reference/resource-configs/target_database.md +++ b/website/docs/reference/resource-configs/target_database.md @@ -6,7 +6,7 @@ datatype: string :::note -For [versionless](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#keep-on-latest-version) dbt Cloud accounts and dbt Core v1.9+, this functionality is no longer utilized. Use the [database](/reference/resource-configs/database) config as an alternative to define a custom database while still respecting the `generate_database_name` macro. +For [versionless](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#versionless) dbt Cloud accounts and dbt Core v1.9+, this functionality is no longer utilized. Use the [database](/reference/resource-configs/database) config as an alternative to define a custom database while still respecting the `generate_database_name` macro. ::: diff --git a/website/docs/reference/resource-configs/target_schema.md b/website/docs/reference/resource-configs/target_schema.md index 8af0daff42d..893686a7513 100644 --- a/website/docs/reference/resource-configs/target_schema.md +++ b/website/docs/reference/resource-configs/target_schema.md @@ -6,7 +6,7 @@ datatype: string :::note -For [versionless](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#keep-on-latest-version) dbt Cloud accounts and dbt Core v1.9+, this functionality is no longer required. Use the [schema](/reference/resource-configs/schema) config as an alternative to define a custom schema while still respecting the `generate_schema_name` macro. +For [versionless](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#versionless) dbt Cloud accounts and dbt Core v1.9+, this functionality is no longer required. Use the [schema](/reference/resource-configs/schema) config as an alternative to define a custom schema while still respecting the `generate_schema_name` macro. ::: @@ -40,9 +40,6 @@ On **BigQuery**, this is analogous to a `dataset`. ## Default This is a **required** parameter, no default is provided. -## FAQs - - ## Examples ### Build all snapshots in a schema named `snapshots` diff --git a/website/docs/reference/resource-properties/unit-tests.md b/website/docs/reference/resource-properties/unit-tests.md index 3c6e6eef0a6..08081c4c24a 100644 --- a/website/docs/reference/resource-properties/unit-tests.md +++ b/website/docs/reference/resource-properties/unit-tests.md @@ -7,7 +7,7 @@ datatype: test :::note -This functionality is only supported in dbt Core v1.8+ or dbt Cloud accounts that have gone versionless by opting to ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). +This functionality is only supported in dbt Core v1.8+ or dbt Cloud accounts that have gone ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless). ::: diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js index 1ee1a308546..9eb040e54fb 100644 --- a/website/docusaurus.config.js +++ b/website/docusaurus.config.js @@ -81,7 +81,7 @@ var siteSettings = { "https://www.getdbt.com/resources/webinars/dbt-cloud-demos-with-experts/?utm_medium=internal&utm_source=docs&utm_campaign=q2-2025_biweekly-demos_aw&utm_content=biweekly-demos____&utm_term=all_all__", // Set community spotlight member on homepage // This is the ID for a specific file under docs/community/spotlight - communitySpotlightMember: "tyler-rouze", + communitySpotlightMember: "meagan-palmer", prism: { theme: (() => { var theme = require("prism-react-renderer/themes/nightOwl"); diff --git a/website/sidebars.js b/website/sidebars.js index 5b4fd6f1bd8..7bb8a43d85b 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -59,6 +59,7 @@ const sidebarSettings = { }, items: [ "docs/cloud/connect-data-platform/about-connections", + "docs/cloud/connect-data-platform/connect-amazon-athena", "docs/cloud/connect-data-platform/connect-azure-synapse-analytics", "docs/cloud/connect-data-platform/connect-microsoft-fabric", "docs/cloud/connect-data-platform/connect-starburst-trino", @@ -79,6 +80,7 @@ const sidebarSettings = { items: [ "docs/cloud/manage-access/about-user-access", "docs/cloud/manage-access/invite-users", + "docs/cloud/manage-access/mfa", { type: "category", label: "User permissions and licenses", @@ -462,6 +464,7 @@ const sidebarSettings = { "docs/deploy/job-scheduler", "docs/deploy/deploy-environments", "docs/deploy/continuous-integration", + "docs/deploy/continuous-deployment", { type: "category", label: "Jobs", diff --git a/website/snippets/_adapters-trusted.md b/website/snippets/_adapters-trusted.md index 178f1e3224b..2ac5268fc28 100644 --- a/website/snippets/_adapters-trusted.md +++ b/website/snippets/_adapters-trusted.md @@ -14,7 +14,7 @@ diff --git a/website/snippets/_cloud-environments-info.md b/website/snippets/_cloud-environments-info.md index 5013f9763ff..166165be855 100644 --- a/website/snippets/_cloud-environments-info.md +++ b/website/snippets/_cloud-environments-info.md @@ -35,7 +35,7 @@ Both development and deployment environments have a section called **General Set - dbt Cloud allows users to select any dbt release. At this time, **environments must use a dbt version greater than or equal to v1.0.0;** [lower versions are no longer supported](/docs/dbt-versions/upgrade-dbt-version-in-cloud). - If you select a current version with `(latest)` in the name, your environment will automatically install the latest stable version of the minor version selected. -- Go versionless by opting to **Keep on latest version**, which removes the need for manually upgrading environment, while ensuring you are always up to date with the latest fixes and features. +- Go **Versionless**, which removes the need for manually upgrading environment, while ensuring you are always up to date with the latest fixes and features. ::: ### Custom branch behavior diff --git a/website/snippets/_config-dbt-version-check.md b/website/snippets/_config-dbt-version-check.md index b5283aae864..d4e495bd379 100644 --- a/website/snippets/_config-dbt-version-check.md +++ b/website/snippets/_config-dbt-version-check.md @@ -1,5 +1,5 @@ -Starting in 2024, when you select **Keep on latest version** in dbt Cloud, dbt will ignore the `require-dbt-version` config. Refer to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) for more details about going versionless. +Starting in 2024, when you select **Versionless** in dbt Cloud, dbt will ignore the `require-dbt-version` config. Refer to [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) for more details. dbt Labs is committed to zero breaking changes for code in dbt projects, with ongoing releases to dbt Cloud and new versions of dbt Core. We also recommend these best practices: diff --git a/website/snippets/_mesh-cycle-detection.md b/website/snippets/_mesh-cycle-detection.md new file mode 100644 index 00000000000..2a48d0a15bd --- /dev/null +++ b/website/snippets/_mesh-cycle-detection.md @@ -0,0 +1,14 @@ +Currently, the default behavior for "project" dependencies enforces that these relationships only go in one direction, meaning that the `jaffle_finance` project could not add a new model that depends, on any public models produced by the `jaffle_marketing` project. dbt will check for cycles across projects and raise errors if any are detected. + +However, many teams may want to be able to share data assets back and forth between teams. _We've added support for enabling bidirectional dependencies across projects, currently in beta_. + +To enable this in your account, set the environment variable `DBT_CLOUD_PROJECT_CYCLES_ALLOWED` to `TRUE` in all your dbt Cloud environments. This allows you to create bidirectional dependencies between projects, so long as the new dependency does not introduce any node-level cycles. + +When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: + +1. The `project_a` project runs in a deployment environment and produces public models. +2. The `project_b` project adds `project_a` as a dependency. +3. The `project_b` project runs in a deployment environment and produces public models. +4. The `project_a` project adds `project_b` as a dependency. + +If you enable this feature and experience any issues, please reach out to [dbt Cloud support](mailto:support@getdbt.com). diff --git a/website/snippets/_sl-deprecation-notice.md b/website/snippets/_sl-deprecation-notice.md deleted file mode 100644 index 2c42dd199c7..00000000000 --- a/website/snippets/_sl-deprecation-notice.md +++ /dev/null @@ -1,5 +0,0 @@ -:::info Deprecation of dbt Metrics and the legacy dbt Semantic Layer -dbt Labs has deprecated dbt Metrics and the legacy dbt Semantic Layer, both supported on dbt version 1.5 or lower. These changes went into effect on December 15th, 2023. - -To migrate and access [MetricFlow](/docs/build/build-metrics-intro) or the re-released dbt Semantic Layer, use the [dbt Semantic Layer migration guide](/guides/sl-migration) and [upgrade your version](/docs/dbt-versions/upgrade-dbt-version-in-cloud) in dbt Cloud. -::: diff --git a/website/snippets/cloud-feature-parity.md b/website/snippets/cloud-feature-parity.md index 6c6f224f215..b6c49aa0faa 100644 --- a/website/snippets/cloud-feature-parity.md +++ b/website/snippets/cloud-feature-parity.md @@ -1,17 +1,17 @@ The following table outlines which dbt Cloud features are supported on the different SaaS options available today. For more information about feature availability, please [contact us](https://www.getdbt.com/contact/). -| Feature | Multi-tenant | AWS single tenant | Azure single tenant | -|-------------------------------|--------------|-----------------------|----------------------| -| Audit logs | ✅ | ✅ | ✅ | -| Continuous integration jobs | ✅ | ✅ | ✅ | -| dbt Cloud CLI | ✅ | ✅ | ✅ | -| dbt Cloud IDE | ✅ | ✅ | ✅ | -| dbt Explorer | ✅ | ✅ | ✅ | -| dbt Mesh | ✅ | ✅ | ✅ | -| dbt Semantic Layer | ✅ | ✅ (Upon request) | ❌ | -| Discovery API | ✅ | ✅ | ✅ | -| IP restrictions | ✅ | ✅ | ✅ | -| Job scheduler | ✅ | ✅ | ✅ | -| PrivateLink egress | ✅ (AWS only)| ✅ | ✅ | -| PrivateLink ingress | ❌ | ✅ | ✅ | -| Webhooks (Outbound) | ✅ | ✅ | ❌ | +| Feature | AWS Multi-tenant | AWS single tenant |Azure multi-tenant ([Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud)) | Azure single tenant | +|-------------------------------|------------------|-----------------------|---------------------|------------------- -| +| Audit logs | ✅ | ✅ | ✅ | ✅ | +| Continuous integration jobs | ✅ | ✅ | ✅ | ✅ | +| dbt Cloud CLI | ✅ | ✅ | ✅ | ✅ | +| dbt Cloud IDE | ✅ | ✅ | ✅ | ✅ | +| dbt Explorer | ✅ | ✅ | ✅ | ✅ | +| dbt Mesh | ✅ | ✅ | ✅ | ✅ | +| dbt Semantic Layer | ✅ | ✅ (Upon request) | ✅ | ❌ | +| Discovery API | ✅ | ✅ | ✅ | ✅ | +| IP restrictions | ✅ | ✅ | ✅ | ✅ | +| Job scheduler | ✅ | ✅ | ✅ | ✅ | +| PrivateLink egress | ✅ (AWS only) | ✅ | ✅ | ✅ | +| PrivateLink ingress | ❌ | ✅ | ❌ | ✅ | +| Webhooks (Outbound) | ✅ | ✅ | ✅ | ❌ | diff --git a/website/snippets/core-versions-table.md b/website/snippets/core-versions-table.md index 13a77f6dd30..b9f79f7d76a 100644 --- a/website/snippets/core-versions-table.md +++ b/website/snippets/core-versions-table.md @@ -4,7 +4,7 @@ |:-------------------------------------------------------------:|:---------------:|:-------------------------------------:| | [**v1.8**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8) | May 9 2024 | Active — May 8, 2025 | | [**v1.7**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.7) | Nov 2, 2023 | Critical — Nov 1, 2024 | -| [**v1.6**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.6) | Jul 31, 2023 | Critical — Jul 30, 2024 | +| [**v1.6**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.6) | Jul 31, 2023 | End of Life* ⚠️ | | [**v1.5**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.5) | Apr 27, 2023 | End of Life* ⚠️ | | [**v1.4**](/docs/dbt-versions/core-upgrade/older%20versions/upgrading-to-v1.4) | Jan 25, 2023 | End of Life* ⚠️ | | [**v1.3**](/docs/dbt-versions/core-upgrade/older%20versions/upgrading-to-v1.3) | Oct 12, 2022 | End of Life* ⚠️ | diff --git a/website/src/components/communitySpotlightList/index.js b/website/src/components/communitySpotlightList/index.js index c52fe7891c7..8355fd0958b 100644 --- a/website/src/components/communitySpotlightList/index.js +++ b/website/src/components/communitySpotlightList/index.js @@ -11,7 +11,7 @@ const communityDescription = "The dbt Community is where analytics engineering l // This date determines where the 'Previously on the Spotlight" text will show. // Any spotlight members with a 'dateCreated' field before this date // will be under the 'Previously..' header. -const currentSpotlightDate = new Date('2024-05-01') +const currentSpotlightDate = new Date('2024-07-26') function CommunitySpotlightList({ spotlightData }) { const { siteConfig } = useDocusaurusContext() diff --git a/website/static/img/community/spotlight/Meagan-Palmer.png b/website/static/img/community/spotlight/Meagan-Palmer.png new file mode 100644 index 00000000000..b24bf8db484 Binary files /dev/null and b/website/static/img/community/spotlight/Meagan-Palmer.png differ diff --git a/website/static/img/community/spotlight/Mikko-Sulonen.png b/website/static/img/community/spotlight/Mikko-Sulonen.png new file mode 100644 index 00000000000..faba53b639b Binary files /dev/null and b/website/static/img/community/spotlight/Mikko-Sulonen.png differ diff --git a/website/static/img/community/spotlight/Radovan-Bacovic.png b/website/static/img/community/spotlight/Radovan-Bacovic.png new file mode 100644 index 00000000000..f4e98bede9d Binary files /dev/null and b/website/static/img/community/spotlight/Radovan-Bacovic.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png new file mode 100644 index 00000000000..8e02ba262fe Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png new file mode 100644 index 00000000000..c4754e4bf60 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png new file mode 100644 index 00000000000..8e02ba262fe Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png new file mode 100644 index 00000000000..c4754e4bf60 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png differ