diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 5e5371c4589..8aaf0375007 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -3,10 +3,6 @@ # Anything unmatched by selectors below * @dbt-labs/product-docs -# The blog -/website/blog @KiraFuruichi - - # Adapter & Package Development Docs /website/docs/docs/supported-data-platforms.md @dbt-labs/product-docs @dataders /website/docs/reference/warehouse-setups @dbt-labs/product-docs @dataders @@ -17,4 +13,6 @@ /website/docs/guides/building-packages @dbt-labs/product-docs @amychen1776 @dataders @dbeatty10 /website/docs/guides/creating-new-materializations @dbt-labs/product-docs @dataders @dbeatty10 - +# Require approval from the Multicell team when making +# changes to the public facing migration documentation. +/website/docs/docs/cloud/migration.md @dbt-labs/dev-multicell diff --git a/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md b/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md index 5098e9d29e3..f7e5786bc70 100644 --- a/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md +++ b/website/blog/2021-11-23-how-to-upgrade-dbt-versions.md @@ -18,7 +18,7 @@ It's been a few years since dbt-core turned 1.0! Since then, we've committed to In 2024, we're taking this promise further by: - Stabilizing interfaces for everyone — adapter maintainers, metadata consumers, and (of course) people writing dbt code everywhere — as discussed in [our November 2023 roadmap update](https://github.com/dbt-labs/dbt-core/blob/main/docs/roadmap/2023-11-dbt-tng.md). -- Introducing **Keep on latest version** in dbt Cloud. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud). +- Introducing **Versionless** in dbt Cloud. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud). We're leaving the rest of this post as is, so we can all remember how it used to be. Enjoy a stroll down memory lane. diff --git a/website/blog/2024-04-22-extended-attributes.md b/website/blog/2024-04-22-extended-attributes.md index d00eb57569d..18d4ff0b64c 100644 --- a/website/blog/2024-04-22-extended-attributes.md +++ b/website/blog/2024-04-22-extended-attributes.md @@ -80,7 +80,7 @@ All you need to do is configure an environment as staging and enable the **Defer ## Upgrading on a curve -Lastly, let’s consider a more specialized use case. Imagine we have a "tiger team" (consisting of a lone analytics engineer named Dave) tasked with upgrading from dbt version 1.6 to the new **Keep on latest version** setting, to take advantage of added stability and feature access. We want to keep the rest of the data team being productive in dbt 1.6 for the time being, while enabling Dave to upgrade and do his work in the new versionless mode. +Lastly, let’s consider a more specialized use case. Imagine we have a "tiger team" (consisting of a lone analytics engineer named Dave) tasked with upgrading from dbt version 1.6 to the new **Versionless** setting, to take advantage of added stability and feature access. We want to keep the rest of the data team being productive in dbt 1.6 for the time being, while enabling Dave to upgrade and do his work in the new versionless mode. ### Development environment diff --git a/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md b/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md index 0953e9b3c27..078dab198fa 100644 --- a/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md +++ b/website/blog/2024-05-22-latest-dbt-stability-improvement-innovation.md @@ -1,5 +1,5 @@ --- -title: "How we're making sure you can confidently \"Keep on latest version\" in dbt Cloud" +title: "How we're making sure you can confidently go \"Versionless\" in dbt Cloud" description: "Over the past 6 months, we've laid a stable foundation for continuously improving dbt." slug: latest-dbt-stability @@ -16,19 +16,19 @@ As long as dbt Cloud has existed, it has required users to select a version of d However, this came at a cost. While bumping a project's dbt version *appeared* as simple as selecting from a dropdown, there was real effort required to test the compatibility of the new version against existing projects, package dependencies, and adapters. On the other hand, putting this off meant foregoing access to new features and bug fixes in dbt. -But no more. Today, we're ready to announce the general availability of a new option in dbt Cloud: [**"Keep on latest version."**](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) +But no more. Today, we're ready to announce the general availability of a new option in dbt Cloud: [**"Versionless."**](https://docs.getdbt.com/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) For customers, this means less maintenance overhead, faster access to bug fixes and features, and more time to focus on what matters most: building trusted data products. This will be our stable foundation for improvement and innovation in dbt Cloud. -But we wanted to go a step beyond just making this option available to you. In this blog post, we aim to shed a little light on the extensive work we've done to ensure that using "Keep on latest version" is a stable, reliable experience for the thousands of customers who rely daily on dbt Cloud. +But we wanted to go a step beyond just making this option available to you. In this blog post, we aim to shed a little light on the extensive work we've done to ensure that using "Versionless" is a stable, reliable experience for the thousands of customers who rely daily on dbt Cloud. ## How we safely deploy dbt upgrades to Cloud We've put in place a rigorous, best-in-class suite of tests and control mechanisms to ensure that all changes to dbt under the hood are fully vetted before they're deployed to customers of dbt Cloud. -This pipeline has in fact been in place since January! It's how we've already been shipping continuous changes to the hundreds of customers who've selected "Keep on latest version" while it's been in Beta and Preview. In that time, this process has enabled us to prevent multiple regressions before they were rolled out to any customers. +This pipeline has in fact been in place since January! It's how we've already been shipping continuous changes to the hundreds of customers who've selected "Versionless" while it's been in Beta and Preview. In that time, this process has enabled us to prevent multiple regressions before they were rolled out to any customers. We're very confident in the robustness of this process**. We also know that we'll need to continue building trust with time.** We're sharing details about this work in the spirit of transparency and to build that trust. @@ -82,9 +82,9 @@ All incidents are retrospected to make sure we not only identify and fix the roo ::: -The outcome of this process is that, when you select "Keep on latest version" in dbt Cloud, the time between an improvement being made to dbt Core and you *safely* getting access to it in your projects is a matter of days — rather than months of waiting for the next dbt Core release, on top of any additional time it may have taken to actually carry out the upgrade. +The outcome of this process is that, when you select "Versionless" in dbt Cloud, the time between an improvement being made to dbt Core and you *safely* getting access to it in your projects is a matter of days — rather than months of waiting for the next dbt Core release, on top of any additional time it may have taken to actually carry out the upgrade. -We’re pleased to say that since the beta launch of “Keep on latest version” in dbt Cloud in March, **we have not had any functional regressions reach customers**, while we’ve also been shipping multiple improvements to dbt functionality every day. This is a foundation that we aim to build on for the foreseeable future. +We’re pleased to say that since the beta launch of “Versionless” in dbt Cloud in March, **we have not had any functional regressions reach customers**, while we’ve also been shipping multiple improvements to dbt functionality every day. This is a foundation that we aim to build on for the foreseeable future. ## Stability as a feature @@ -98,7 +98,7 @@ The adapter interface — i.e. how dbt Core actually connects to a third-party d To solve that, we've released a new set of interfaces that are entirely independent of the `dbt-core` library: [`dbt-adapters==1.0.0`](https://github.com/dbt-labs/dbt-adapters). From now on, any changes to `dbt-adapters` will be backward and forward-compatible. This also decouples adapter maintenance from the regular release cadence of dbt Core — meaning maintainers get full control over when they ship implementations of new adapter-powered features. -Note that adapters running in dbt Cloud **must** be [migrated to the new decoupled architecture](https://github.com/dbt-labs/dbt-adapters/discussions/87) as a baseline in order to support the new "Keep on latest version". +Note that adapters running in dbt Cloud **must** be [migrated to the new decoupled architecture](https://github.com/dbt-labs/dbt-adapters/discussions/87) as a baseline in order to support the new "Versionless" option. ### Managing behavior changes: stability as a feature @@ -118,7 +118,7 @@ We’ve now [formalized our development best practices](https://github.com/dbt-l In conclusion, we’re putting a lot of new muscle behind our commitments to dbt Cloud customers, the dbt Community, and the broader ecosystem: -- **Continuous updates**: "Keep on latest version" in dbt Cloud simplifies the update process, ensuring you always have the latest features and bug fixes without the maintenance overhead. +- **Continuous updates**: "Versionless" dbt Cloud simplifies the update process, ensuring you always have the latest features and bug fixes without the maintenance overhead. - **A rigorous new testing and deployment process**: Our new testing pipeline ensures that every update is carefully vetted against documented interfaces, Cloud-supported adapters, and popular packages before it reaches you. This process minimizes the risk of regressions — and has now been successful at entirely preventing them for hundreds of customers over multiple months. - **A commitment to stability**: We’ve reworked our approaches to adapter interfaces, behaviour change management, and metadata artifacts to give you more stability and control. diff --git a/website/blog/2024-06-12-putting-your-dag-on-the-internet.md b/website/blog/2024-06-12-putting-your-dag-on-the-internet.md index 8d0bc79e35d..535cfc34d6e 100644 --- a/website/blog/2024-06-12-putting-your-dag-on-the-internet.md +++ b/website/blog/2024-06-12-putting-your-dag-on-the-internet.md @@ -114,6 +114,6 @@ Traditionally dbt is the T in ELT (dbt overview [here](https://docs.getdbt.com/t In order to get this functionality shipped quickly, EQT opened a pull request, Snowflake helped with some problems we had with CI and a member of dbt Labs helped write the tests and merge the code in! -dbt now features this functionality in dbt 1.8+ or on “Keep on latest version” option of dbt Cloud (dbt overview [here](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)). +dbt now features this functionality in dbt 1.8+ or the “Versionless” option of dbt Cloud (dbt overview [here](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless)). dbt Labs staff and community members would love to chat more about it in the [#db-snowflake](https://getdbt.slack.com/archives/CJN7XRF1B) slack channel. diff --git a/website/blog/authors.yml b/website/blog/authors.yml index 64727801478..85f05a545f9 100644 --- a/website/blog/authors.yml +++ b/website/blog/authors.yml @@ -1,3 +1,4 @@ +--- amy_chen: image_url: /img/blog/authors/achen.png job_title: Product Ecosystem Manager @@ -6,7 +7,14 @@ amy_chen: url: https://www.linkedin.com/in/yuanamychen/ name: Amy Chen organization: dbt Labs - +anders_swanson: + image_url: /img/blog/authors/anders-rain.jpg + job_title: Senior Developer Experience Advocate + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/andersswanson + name: Anders Swanson + organization: dbt Labs andrew_escay: image_url: /img/blog/authors/aescay.jpeg job_title: Analytics Engineer @@ -15,14 +23,12 @@ andrew_escay: url: https://twitter.com/_aescay name: Andrew Escay organization: dbt Labs - arthur_marcon: + description: Arthur Marcon is an Analytics Engineer at Indicium, helping data-driven companies transition to the modern data analytics stack in the cloud through dbt. Arthur is also an enthusiast of technology and innovation. He is mainly interested in how data plays a key role to enable companies to achieve their full innovative and technological potential. Outside of work, he is a movie aficionado, and likes to hit the gym and read Agatha Christie's novels. image_url: /img/blog/authors/arthur-marcon.png job_title: Analytics Engineer name: Arthur Marcon organization: Indicium Tech - description: Arthur Marcon is an Analytics Engineer at Indicium, helping data-driven companies transition to the modern data analytics stack in the cloud through dbt. Arthur is also an enthusiast of technology and innovation. He is mainly interested in how data plays a key role to enable companies to achieve their full innovative and technological potential. Outside of work, he is a movie aficionado, and likes to hit the gym and read Agatha Christie's novels. - barr_yaron: image_url: /img/blog/authors/barr-yaron.png job_title: Product Manager @@ -31,7 +37,6 @@ barr_yaron: url: https://www.linkedin.com/in/barryaron/ name: Barr Yaron organization: dbt Labs - bennie_regenold: image_url: /img/blog/authors/bennie-regenold.png job_title: Analytics Engineer @@ -40,7 +45,6 @@ bennie_regenold: url: https://www.linkedin.com/in/benjaminregenold/ name: Bennie Regenold organization: dbt Labs - benoit_perigaud: image_url: /img/blog/authors/benoit-perigaud.jpeg job_title: Staff Analytics Engineer @@ -49,7 +53,6 @@ benoit_perigaud: url: https://www.linkedin.com/in/benoit-perigaud/ name: Benoit Perigaud organization: dbt Labs - brittany_krauth: description: Brittany Krauth works as Manager, Analytics & Insights supporting Degreed's upskilling platform. Brittany is passionate about building a company-wide data-driven culture. She has worked in various analytical roles, from a focus on predictive analytics to data visualization to process improvements. In addition, she holds a BS in Industrial Engineering from Georgia Tech. In her spare time, Brittany competes in dog agility and trains donkeys. image_url: /img/blog/authors/brittany-krauth.png @@ -59,7 +62,15 @@ brittany_krauth: url: https://www.linkedin.com/in/brittanykrauth name: Brittany Krauth organization: Degreed - +callie_white: + description: >- + Callie White is a Senior Analytics Consultant at Montreal Analytics where she leads projects in partnership with client data teams to achieve the best-practice and best-suited results. She’s also passionate about mentoring developers and demystifying the complexity of a role in data. Callie uses data-driven decision-making in everything she does; mainly to maximize cuddle time with her rescue pup named Snacks, or to scope out her next hiking trip. Montreal Analytics is a full-stack data + consultancy servicing North America, and are both a dbt Preferred Consulting Partner and Platinum dbt Certification Award winner. We help organizations on the whole data journey, from strategic advisory to hands-on development and enablement. Whether deploying greenfield data platforms, tackling complex migrations or auditing & refactoring entangled data models, we build scalable solutions for organizations of all sizes. If you are interested in learning more about how we help clients across + the modern stack, contact us today. + image_url: /img/blog/authors/callie-white.jpg + job_title: Analytics Consultant + name: Callie White + organization: Montreal Analytics callum_mccann: description: Callum works on metrics and is either talking about that or obsessing about his dog. image_url: /img/blog/authors/callum-mccann.jpg @@ -71,14 +82,6 @@ callum_mccann: url: https://twitter.com/callumpmccann name: Callum McCann organization: dbt Labs - -callie_white: - description: Callie White is a Senior Analytics Consultant at Montreal Analytics where she leads projects in partnership with client data teams to achieve the best-practice and best-suited results. She’s also passionate about mentoring developers and demystifying the complexity of a role in data. Callie uses data-driven decision-making in everything she does; mainly to maximize cuddle time with her rescue pup named Snacks, or to scope out her next hiking trip. Montreal Analytics is a full-stack data consultancy servicing North America, and are both a dbt Preferred Consulting Partner and Platinum dbt Certification Award winner. We help organizations on the whole data journey, from strategic advisory to hands-on development and enablement. Whether deploying greenfield data platforms, tackling complex migrations or auditing & refactoring entangled data models, we build scalable solutions for organizations of all sizes. If you are interested in learning more about how we help clients across the modern stack, contact us today. - image_url: /img/blog/authors/callie-white.jpg - job_title: Analytics Consultant - name: Callie White - organization: Montreal Analytics - charlie_summers: description: Charlie is the Data Engineer Tech Lead at Merit. He introduced Merit to dbt and it's been a fantastic fit for a wide variety of data pipelines. He likes thinking about the future of data - integrating event streams, analyzing encrypted data, capturing fine-grained lineage, and making it easy to build simple apps on top of data warehouses/lakes. image_url: /img/blog/authors/charlie-summers.jpeg @@ -88,26 +91,30 @@ charlie_summers: url: https://www.linkedin.com/in/charliesummers name: Charlie Summers organization: Merit - +chenyu_li: + image_url: /img/blog/authors/chenyu-li.png + job_title: Staff Software Engineer + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/chenyu-li-450b06a8 + name: Chenyu Li + organization: dbt Labs christian_vanbellen: + description: Christian van Bellen is an Analytics Engineer at Indicium Tech, acting mainly in projects focused on Enterprise-level data stack transition. Also works with the DataScience/ML stack. Christian started using dbt in 2022 to help implement a modern data stack within enterprises, and is a certified dbt developer. image_url: /img/blog/authors/christian-van-bellen.JPG job_title: Analytics Engineer name: Christian van Bellen organization: Indicium Tech - description: Christian van Bellen is an Analytics Engineer at Indicium Tech, acting mainly in projects focused on Enterprise-level data stack transition. Also works with the DataScience/ML stack. Christian started using dbt in 2022 to help implement a modern data stack within enterprises, and is a certified dbt developer. - christine_berger: image_url: /img/blog/authors/christine-berger.jpeg job_title: Senior Analytics Engineer name: Christine Berger organization: dbt Labs - christophe_oudar: image_url: /img/blog/authors/christophe-oudar.png job_title: Staff Software Engineer name: Christophe Oudar organization: Teads - claire_carroll: image_url: /img/blog/authors/claire.jpeg job_title: Analytics Engineer @@ -120,7 +127,14 @@ claire_carroll: url: https://www.linkedin.com/in/clrcrl/ name: Claire Carroll organization: analyticsengineers.club - +colin_rogers: + image_url: /img/blog/authors/colin-rogers.jpeg + job_title: Senior Software Engineer + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/ckrogers + name: Colin Rogers + organization: dbt Labs dave_connors: image_url: /img/blog/authors/dconnors.jpeg job_title: Senior Developer Experience Advocate @@ -131,7 +145,6 @@ dave_connors: url: https://www.linkedin.com/in/daveconnors3/ name: Dave Connors organization: dbt Labs - david_krevitt: description: Laziness is a virtue image_url: /img/blog/authors/dkrevitt.jpg @@ -141,7 +154,6 @@ david_krevitt: url: https://twitter.com/dkrevitt name: David Krevitt organization: dbt Labs - donny_flynn: description: Donny Flynn is a customer data architect at Census - a reverse ETL product that helps companies operationalize analytics. Prior to joining Census, Donny built out the data stack at Owner and led a data team at Chiper. When not doing data work, he's most likely watching Chicago sports. image_url: /img/blog/authors/dflynn.jpeg @@ -153,13 +165,11 @@ donny_flynn: url: https://www.linkedin.com/in/donny-flynn-578149a4/ name: Donny Flynn organization: Census - doug_beatty: image_url: /img/blog/authors/dbeatty.jpeg job_title: Senior Developer Experience Advocate name: Doug Beatty organization: dbt Labs - doug_guthrie: image_url: /img/blog/authors/dguthrie.jpeg job_title: Senior Solutions Architect @@ -170,13 +180,12 @@ doug_guthrie: url: https://www.linkedin.com/in/doug-guthrie-07994a48/ name: Doug Guthrie organization: dbt Labs - emily_riederer: - description: Emily Riederer is a Senior Analytics Manager at Capital One where she leads a team delivering a portfolio of data products, enterprise analysis tools, and data science solutions to business partners. As part of the dbt community, she develops and maintains the dbtplyr package. In her spare time, Emily frequently writes about data (see emilyriederer.com, The R Markdown Cookbook, and 97 Things Every Data Engineer Should Know), reviews technical manuscripts for CRC Press, and supports open research software engineering as a member of the rOpenSci editorial board. + description: >- + Emily Riederer is a Senior Analytics Manager at Capital One where she leads a team delivering a portfolio of data products, enterprise analysis tools, and data science solutions to business partners. As part of the dbt community, she develops and maintains the dbtplyr package. In her spare time, Emily frequently writes about data (see emilyriederer.com, The R Markdown Cookbook, and 97 Things Every Data Engineer Should Know), reviews technical manuscripts for CRC Press, and supports open + research software engineering as a member of the rOpenSci editorial board. image_url: /img/blog/authors/emily-riederer.jpeg job_title: Senior Manager Analytics - name: Emily Riederer - organization: Capital One links: - icon: fa-linkedin url: https://www.linkedin.com/in/emilyriederer/ @@ -186,17 +195,33 @@ emily_riederer: url: https://twitter.com/emilyriederer - icon: fa-readme url: https://emilyriederer.com - + name: Emily Riederer + organization: Capital One +ernesto_ongaro: + image_url: /img/blog/authors/ernesto-ongaro.png + job_title: Senior Solutions Architect + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/eongaro + name: Ernesto Ongaro + organization: dbt Labs euan_johnston: image_url: /img/blog/authors/ejohnston.png job_title: Freelance Business Intelligence manager - name: Euan Johnston links: - icon: fa-linkedin url: https://www.linkedin.com/in/euan-johnston-610a05a8/ - icon: fa-github url: https://github.com/euanjohnston-dev - + name: Euan Johnston +filip_byrén: + image_url: /img/blog/authors/filip-eqt.png + job_title: VP and Software Architect + links: + - icon: fa-linked + url: https://www.linkedin.com/in/filip-byr%C3%A9n/ + name: Filip Byrén + organization: EQT Group grace_goheen: image_url: /img/blog/authors/grace-goheen.jpeg job_title: Analytics Engineer @@ -205,9 +230,7 @@ grace_goheen: url: https://www.linkedin.com/in/gracegoheen/ name: Grace Goheen organization: dbt Labs - gwen_windflower: - name: Gwen Windflower image_url: /img/blog/authors/gwen-windflower.jpeg job_title: Senior Developer Experience Advocate links: @@ -215,7 +238,7 @@ gwen_windflower: url: https://www.linkedin.com/in/gwenwindflower - icon: fa-github url: https://github.com/gwenwindflower - + name: Gwen Windflower ian_fahey: image_url: /img/blog/authors/ian-fahey.png job_title: Analytics Engineer @@ -226,13 +249,19 @@ ian_fahey: url: https://twitter.com/Cavorax name: Ian Fahey organization: dbt Labs - izzy_erekson: image_url: /img/blog/authors/izzy.jpeg job_title: Solutions Architect name: Izzy Erekson organization: dbt Labs - +jade_milaney: + description: >- + Jade Milaney is an Analytics Consultant at Montreal Analytics where in with client data teams to implement data models and data engineering solutions. Jade studied mathematics in university, and has an interest in all things statistics and analytics. Montreal Analytics is a full-stack data consultancy servicing North America, and are both a dbt Preferred Consulting Partner and Platinum dbt Certification Award winner. We help organizations on the whole data journey, from strategic advisory to + hands-on development and enablement. Whether deploying greenfield data platforms, tackling complex migrations or auditing & refactoring entangled data models, we build scalable solutions for organizations of all sizes. If you are interested in learning more about how we help clients across the modern stack, contact us today. + image_url: /img/blog/authors/jade-milaney.jpg + job_title: Analytics Consultant + name: Jade Milaney + organization: Montreal Analytics jason_ganz: image_url: /img/blog/authors/jasnonaz.jpeg job_title: Developer Experience @@ -241,20 +270,11 @@ jason_ganz: url: https://twitter.com/jasnonaz name: Jason Ganz organization: dbt Labs - -jade_milaney: - description: Jade Milaney is an Analytics Consultant at Montreal Analytics where in with client data teams to implement data models and data engineering solutions. Jade studied mathematics in university, and has an interest in all things statistics and analytics. Montreal Analytics is a full-stack data consultancy servicing North America, and are both a dbt Preferred Consulting Partner and Platinum dbt Certification Award winner. We help organizations on the whole data journey, from strategic advisory to hands-on development and enablement. Whether deploying greenfield data platforms, tackling complex migrations or auditing & refactoring entangled data models, we build scalable solutions for organizations of all sizes. If you are interested in learning more about how we help clients across the modern stack, contact us today. - image_url: /img/blog/authors/jade-milaney.jpg - job_title: Analytics Consultant - name: Jade Milaney - organization: Montreal Analytics - jeremy_cohen: image_url: /img/blog/authors/jerco.jpeg job_title: Product Manager name: Jeremy Cohen organization: dbt Labs - jess_williams: image_url: /img/blog/authors/jess.png job_title: Head of Professional Services @@ -263,7 +283,6 @@ jess_williams: url: https://www.linkedin.com/in/jessdwilliams/ name: Jess Williams organization: dbt Labs - joao_antunes: image_url: /img/blog/authors/joao_antunes.jpg job_title: Lead Data Engineer, Global Product Strategy @@ -272,7 +291,6 @@ joao_antunes: url: https://www.linkedin.com/in/jo%C3%A3o-bernardo-pires-antunes name: João Antunes organization: Roche - joe_markiewicz: description: Joe is a dbt package maintainer/manager at Fivetran by day, and a cat obsessed video game developer by night. Actually, Joe is cat obsessed all day. image_url: /img/blog/authors/joe-markiewicz.jpeg @@ -284,7 +302,6 @@ joe_markiewicz: url: https://twitter.com/JoeMarkiewicz17 name: Joe Markiewicz organization: Fivetran - joel_labes: image_url: /img/blog/authors/jlabes.png job_title: Senior Developer Experience Advocate @@ -295,7 +312,6 @@ joel_labes: url: https://twitter.com/joellabes name: Joel Labes organization: dbt Labs - jonathan_natkins: description: Natty also writes about startups, equity, data, and more in his Substack called [Semi-Structured](http://semistructured.substack.com/). image_url: /img/blog/authors/jonathan-natkins.jpeg @@ -307,6 +323,15 @@ jonathan_natkins: url: https://twitter.com/nattyice name: Jon "Natty" Natkins organization: dbt Labs +jonathan_neo: + description: Jonathan is a Data Engineer at Canva where he is building and maintaining petabyte-scale data platforms. Jonathan founded Data Engineer Camp, a bootcamp that empowers professionals to become proficient data engineers. He has since trained data and software professionals from around that are working at companies like Microsoft, Atlassian, and Apple. + image_url: /img/blog/authors/jonathan-neo.png + job_title: Data Engineer + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/jonneo/ + name: Jonathan Neo + organization: Canva & Data Engineer Camp jordan_stein: image_url: /img/blog/authors/jordan.jpeg job_title: Product Manager @@ -315,7 +340,6 @@ jordan_stein: url: https://www.linkedin.com/in/jstein5/ name: Jordan Stein organization: dbt Labs - josh_fell: image_url: /img/blog/authors/josh-fell.jpeg job_title: Ecosystem Engineer @@ -324,7 +348,6 @@ josh_fell: url: https://www.linkedin.com/in/josh-fell/ name: Josh Fell organization: Astronomer - kira_furuichi: image_url: /img/blog/authors/kira-furuichi.png job_title: Technical Writer @@ -333,7 +356,6 @@ kira_furuichi: url: https://www.linkedin.com/in/kira-furuichi/ name: Kira Furuichi organization: dbt Labs - kshitij_aranke: image_url: /img/blog/authors/kshitij-aranke.jpg job_title: Senior Software Engineer @@ -342,7 +364,6 @@ kshitij_aranke: url: https://www.linkedin.com/in/aranke/ name: Kshitij Aranke organization: dbt Labs - lauren_benezra: image_url: /img/blog/authors/lauren-benezra.jpeg job_title: Analytics Engineer @@ -351,7 +372,6 @@ lauren_benezra: url: https://www.linkedin.com/in/lbenezra/ name: Lauren Benezra organization: dbt Labs - lauren_craigie: image_url: /img/blog/authors/craigie.png job_title: Product Marketing @@ -360,14 +380,12 @@ lauren_craigie: url: https://twitter.com/crai_fish name: Lauren Craigie organization: dbt Labs - lucas_bergodias: + description: Lucas Dias works as an Analytics Engineer at Indicium Tech, a leading technology consultancy that specializes in data-driven solutions. Lucas brings a wealth of expertise and experience to the role, having worked in the analytics field for two years. At Indicium, he assists companies in transitioning to modern data stack through the use of dbt. He's passionate about using data to drive innovation and growth, and primarily works on enterprise-level projects. image_url: /img/blog/authors/lucas-dias.jpg job_title: Analytics Engineer name: Lucas Bergo Dias organization: Indicium Tech - description: Lucas Dias works as an Analytics Engineer at Indicium Tech, a leading technology consultancy that specializes in data-driven solutions. Lucas brings a wealth of expertise and experience to the role, having worked in the analytics field for two years. At Indicium, he assists companies in transitioning to modern data stack through the use of dbt. He's passionate about using data to drive innovation and growth, and primarily works on enterprise-level projects. - matt_winkler: description: Matt is an ex-data scientist who chose to embrace the simplicity of using SQL to manage and testing data pipelines with dbt. He previously worked as a hands-on ML practitioner, and consulted with Fortune 500 clients to build and maintain ML Ops pipelines using (mostly) AWS Sagemaker. He lives in the Denver area, and you can say hello on dbt Slack or on LinkedIn. image_url: /img/blog/authors/matt-winkler.jpeg @@ -376,13 +394,19 @@ matt_winkler: - url: https://www.linkedin.com/in/matt-winkler-4024263a/ name: Matt Winkler organization: dbt Labs - +michelle_ark: + image_url: /img/blog/authors/michelle-ark.png + job_title: Staff Software Engineer + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/michelle-ark + name: Michelle Ark + organization: dbt Labs mikael_thorup: image_url: /img/blog/authors/mikael-thorup.jpeg job_title: Senior Analytics Engineer name: Mikael Thorup organization: Lunar - nate_sooter: image_url: /img/blog/authors/nate-sooter.jpeg job_title: Manager of BI Operations @@ -393,14 +417,22 @@ nate_sooter: url: https://www.linkedin.com/in/nathansooter name: Nate Sooter organization: Smartsheet - +nathaniel_burren: + image_url: /img/blog/authors/nathaniel_burren.jpeg + job_title: Senior Analytics Engineer + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/nathaniel-burren-6037783b/ + name: Nathaniel Burren + organization: Cityblock Health noah_kennedy: + description: >- + Noah Kennedy is a Software Developer at Tempus Labs, helping manage the worlds largest repository of clinical and molecular patient data. Noah picked up dbt at Tempus 2 years ago, and has worked extensively on internal dbt development for Tempus — creating macros, custom materializations, and testing frameworks that allow scientists to experiment more and data engineer less. Outside of Tempus, Noah spends his time running, biking, playing music, and exploring coffee shops. You can find more + of his writing at https://noahkennedy.co/about/. image_url: /img/blog/authors/noah-kennedy.png job_title: Software Developer - description: Noah Kennedy is a Software Developer at Tempus Labs, helping manage the worlds largest repository of clinical and molecular patient data. Noah picked up dbt at Tempus 2 years ago, and has worked extensively on internal dbt development for Tempus — creating macros, custom materializations, and testing frameworks that allow scientists to experiment more and data engineer less. Outside of Tempus, Noah spends his time running, biking, playing music, and exploring coffee shops. You can find more of his writing at https://noahkennedy.co/about/. - organization: Tempus name: Noah Kennedy - + organization: Tempus pat_kearns: image_url: /img/blog/authors/pkearns.png job_title: Senior Analytics Engineer @@ -409,7 +441,6 @@ pat_kearns: url: https://www.linkedin.com/in/pat-kearns/ name: Pat Kearns organization: dbt Labs - pedro_brito_de_sa: image_url: /img/blog/authors/pedro_brito.jpeg job_title: Product Analyst @@ -418,7 +449,6 @@ pedro_brito_de_sa: url: https://www.linkedin.com/in/pbritosa/ name: Pedro Brito de Sa organization: Sage - rastislav_zdechovan: image_url: /img/blog/authors/rastislav-zdechovan.png job_title: Analytics Engineer @@ -427,7 +457,6 @@ rastislav_zdechovan: url: https://www.linkedin.com/in/rastislav-zdechovan/ name: Rastislav Zdechovan organization: Infinite Lambda - ross_turk: image_url: /img/blog/authors/ross-turk.png job_title: VP Marketing @@ -438,7 +467,6 @@ ross_turk: url: https://github.com/rossturk name: Ross Turk organization: Datakin - roxi_dahlke: image_url: /img/blog/authors/roxi-dahlke.png job_title: Product Manager @@ -447,14 +475,12 @@ roxi_dahlke: url: https://www.linkedin.com/in/roxanna-dahlke-pourzand-445a3035 name: Roxi Dahlke organization: dbt Labs - sam_harting: description: Sam is a graduate of the inaugural Foundry Program. When Sam isn't helping clients build out their project, he is usually playing video games or making bad jokes to his friends, family, and coworkers. image_url: /img/blog/authors/sam-harting.png job_title: Associate Analytics Engineer name: Samuel Harting organization: dbt Labs - sanjana_sen: image_url: /img/blog/authors/sanjana.png job_title: Analytics Engineer @@ -463,7 +489,14 @@ sanjana_sen: url: https://twitter.com/sqlsanj name: Sanjana Sen organization: dbt Labs - +santiago_jauregui: + image_url: /img/blog/authors/santiago-jauregui.jpeg + job_title: Data Analytics Leader + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/santiago-jauregui/ + name: Santiago Jauregui + organization: MODO sean_mcintyre: image_url: /img/blog/authors/sean_mcintyre.jpg job_title: Senior Solutions Architect @@ -472,7 +505,14 @@ sean_mcintyre: url: https://www.linkedin.com/in/boxysean name: Sean McIntyre organization: dbt Labs - +sebastian_stan: + image_url: /img/blog/authors/sebastian-eqt.png + job_title: Data Engineer + links: + - icon: fa-linkedin + url: https://www.linkedin.com/in/sebastian-lindblom/ + name: Sebastian Stan + organization: EQT Group seth_rosen: description: Seth Rosen is co-founder and CEO of TopCoat Data - a platform for helping organizations build analytical applications. Prior to founding TopCoat, Seth helped companies of all sizes build custom data apps on top of the modern data stack through his consultancy, Hashpath. When he’s not tweeting and thinking about data, he’s tweeting while parenting two toddlers. image_url: /img/blog/authors/seth-rosen.jpeg @@ -484,7 +524,6 @@ seth_rosen: url: https://www.linkedin.com/in/sdrosen/ name: Seth Rosen organization: TopCoat Data - simo_tumelius: image_url: /img/blog/authors/simo-tumelius.jpeg job_title: Freelance Data and Analytics Engineer @@ -492,7 +531,6 @@ simo_tumelius: - icon: fa-linkedin url: https://www.linkedin.com/in/simo-tumelius-00a27a162/ name: Simo Tumelius - simon_podhajsky: description: | Simon Podhajsky is a lapsed neuroscientist turned data everything at iLife Technologies, a startup that seeks to digitize the life insurance agency. @@ -505,7 +543,6 @@ simon_podhajsky: url: https://twitter.com/sim_pod name: Simon Podhajsky organization: iLife Technologies - sterling_paramore: description: | Sterling Paramore started his career in theoretical and computation biophysics and learned that working with data was way more fun than being locked in the ivory tower. He loves solving data engineering and data analytics problems and has been a long time evangelist for dbt. @@ -516,7 +553,6 @@ sterling_paramore: url: https://www.linkedin.com/in/sterlingparamore/ name: Sterling Paramore organization: Mainspring Energy - sung_chung: image_url: /img/blog/authors/sung.jpeg job_title: Solutions Architect @@ -525,7 +561,6 @@ sung_chung: url: https://www.linkedin.com/in/sungwonchung1/ name: Sung Won Chung organization: dbt Labs - viraj_parekh: image_url: /img/blog/authors/viraj-parekh.jpeg job_title: Field CTO @@ -534,14 +569,12 @@ viraj_parekh: url: https://www.linkedin.com/in/viraj-parekh-46114689/ name: Viraj Parekh organization: Astronomer - wasila_quader: description: After a winding road through healthcare spreadsheets and data science projects, Wasila discovered analytics engineering as an apprentice of dbt Labs' Foundry Program. She now works as an analytics engineer on dbt Labs' professional services team. image_url: /img/blog/authors/wasila-quader.png job_title: Associate Analytics Engineer name: Wasila Quader organization: dbt Labs - yannick_misteli: image_url: /img/blog/authors/yannick_misteli.jpg job_title: Head of Engineering, Global Product Strategy @@ -550,7 +583,6 @@ yannick_misteli: url: https://www.linkedin.com/in/yannick-misteli-b86129105/ name: Yannick Misteli organization: Roche - yu_ishikawa: image_url: /img/blog/authors/yu-ishikawa.jpg job_title: Senior Data Privacy Engineer @@ -558,86 +590,4 @@ yu_ishikawa: - icon: fa-linkedin url: https://www.linkedin.com/in/yuishikawa0301 name: Yu Ishikawa - organization: Ubie - -jonathan_neo: - image_url: /img/blog/authors/jonathan-neo.png - job_title: Data Engineer - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/jonneo/ - name: Jonathan Neo - organization: Canva & Data Engineer Camp - description: Jonathan is a Data Engineer at Canva where he is building and maintaining petabyte-scale data platforms. Jonathan founded Data Engineer Camp, a bootcamp that empowers professionals to become proficient data engineers. He has since trained data and software professionals from around that are working at companies like Microsoft, Atlassian, and Apple. - -santiago_jauregui: - image_url: /img/blog/authors/santiago-jauregui.jpeg - job_title: Data Analytics Leader - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/santiago-jauregui/ - name: Santiago Jauregui - organization: MODO - -michelle_ark: - image_url: /img/blog/authors/michelle-ark.png - job_title: Staff Software Engineer - name: Michelle Ark - organization: dbt Labs - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/michelle-ark - -chenyu_li: - image_url: /img/blog/authors/chenyu-li.png - job_title: Staff Software Engineer - name: Chenyu Li - organization: dbt Labs - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/chenyu-li-450b06a8 - -colin_rogers: - image_url: /img/blog/authors/colin-rogers.jpeg - job_title: Senior Software Engineer - name: Colin Rogers - organization: dbt Labs - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/ckrogers - -anders_swanson: - image_url: /img/blog/authors/anders-rain.jpg - job_title: Senior Developer Experience Advocate - name: Anders Swanson - organization: dbt Labs - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/andersswanson - -ernesto_ongaro: - image_url: /img/blog/authors/ernesto-ongaro.png - job_title: Senior Solutions Architect - name: Ernesto Ongaro - organization: dbt Labs - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/eongaro - -sebastian_stan: - image_url: /img/blog/authors/sebastian-eqt.png - job_title: Data Engineer - name: Sebastian Stan - organization: EQT Group - links: - - icon: fa-linkedin - url: https://www.linkedin.com/in/sebastian-lindblom/ - -filip_byrén: - image_url: /img/blog/authors/filip-eqt.png - job_title: VP and Software Architect - name: Filip Byrén - organization: EQT Group - links: - - icon: fa-linked - url: https://www.linkedin.com/in/filip-byr%C3%A9n/ + organization: Ubie \ No newline at end of file diff --git a/website/dbt-versions.js b/website/dbt-versions.js index 0dfe858928f..5ad3a3048c5 100644 --- a/website/dbt-versions.js +++ b/website/dbt-versions.js @@ -15,13 +15,17 @@ exports.versions = [ version: "1.6", EOLDate: "2024-07-31", }, - { - version: "1.5", - EOLDate: "2024-04-27", - }, ] exports.versionedPages = [ + { + "page": "/reference/resource-configs/target_database", + "lastVersion": "1.8", + }, + { + "page": "/reference/resource-configs/target_schema", + "lastVersion": "1.8", + }, { "page": "reference/global-configs/indirect-selection", "firstVersion": "1.8", diff --git a/website/docs/best-practices/best-practice-workflows.md b/website/docs/best-practices/best-practice-workflows.md index 4381906361e..1adae6b2fce 100644 --- a/website/docs/best-practices/best-practice-workflows.md +++ b/website/docs/best-practices/best-practice-workflows.md @@ -8,7 +8,7 @@ This page contains the collective wisdom of experienced users of dbt on how to b ## Best practice workflows ### Version control your dbt project -All dbt projects should be managed in version control. Git branches should be created to manage development of new features and bug fixes. All code changes should be reviewed by a colleague (or yourself) in a Pull Request prior to merging into `master`. +All dbt projects should be managed in version control. Git branches should be created to manage development of new features and bug fixes. All code changes should be reviewed by a colleague (or yourself) in a Pull Request prior to merging into your production branch, such as `main`. :::info Git guide diff --git a/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md b/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md index e57ed5c966f..b6fadc2d7a6 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md +++ b/website/docs/best-practices/how-we-mesh/mesh-2-who-is-dbt-mesh-for.md @@ -20,8 +20,12 @@ Is dbt Mesh a good fit in this scenario? Absolutely! There is no other way to sh ### Adoption challenges -- Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. dbt Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that dbt Mesh will help your teams deliver concrete value quickly. -- Bi-directional project dependencies -- currently, projects in dbt Mesh are treated like dbt resources in that they cannot depend on each other. In reality, domain teams likely need the ability to have “chatty” APIs; otherwise, they need to split projects beyond the 1:1 mapping with team boundaries. While this constraint exists today, we're working to remove this point of friction. More information about this will be provided in the near future! +- Onboarding hundreds of people and dozens of projects is full of friction! The challenges of a scaled, global organization are not to be underestimated. To start the migration, prioritize teams that have strong dbt familiarity and fundamentals. dbt Mesh is an advancement of core dbt deployments, so these teams are likely to have a smoother transition. + + Additionally, prioritize teams that manage strategic data assets that need to be shared widely. This ensures that dbt Mesh will help your teams deliver concrete value quickly. +- Bi-directional project dependencies -- currently, projects in dbt Mesh are treated like dbt resources in that they cannot depend on each other. However, many teams may want to be able to share data assets back and forth between teams. + + We've added support for [enabling bidirectional dependencies](/best-practices/how-we-mesh/mesh-3-structures#cycle-detection) across projects. If this sounds like your organization, dbt Mesh is the architecture you should pursue. ✅ diff --git a/website/docs/best-practices/how-we-mesh/mesh-3-structures.md b/website/docs/best-practices/how-we-mesh/mesh-3-structures.md index 91090521e9f..c75c566610b 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-3-structures.md +++ b/website/docs/best-practices/how-we-mesh/mesh-3-structures.md @@ -18,14 +18,6 @@ At a high level, you’ll need to decide: - Where to draw the lines between your dbt Projects -- i.e. how do you determine where to split your DAG and which models go in which project? - How to manage your code -- do you want multiple dbt Projects living in the same repository (mono-repo) or do you want to have multiple repos with one repo per project? -### Cycle detection - -Like resource dependencies, project dependencies are acyclic, meaning they only move in one direction. This prevents `ref` cycles (or loops), which lead to issues with your data workflows. For example, if project B depends on project A, a new model in project A could not import and use a public model from project B. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies#how-to-write-cross-project-ref) for more information. - -:::note -This requirement is likely to change in the future, so stay tuned for updates! -::: - ## Define your project interfaces by splitting your DAG The first (and perhaps most difficult!) decision when migrating to a multi-project architecture is deciding where to draw the line in your DAG to define the interfaces between your projects. Let's explore some language for discussing the design of these patterns. @@ -74,6 +66,13 @@ Since the launch of dbt Mesh, the most common pattern we've seen is one where pr Users may need to contribute models across multiple projects and this is fine. There will be some friction doing this, versus a single repo, but this is _useful_ friction, especially if upstreaming a change from a “spoke” to a “hub.” This should be treated like making an API change, one that the other team will be living with for some time to come. You should be concerned if your teammates find they need to make a coordinated change across multiple projects very frequently (every week), or as a key prerequisite for ~20%+ of their work. +### Cycle detection + +import CycleDetection from '/snippets/_mesh-cycle-detection.md'; + + + + ### Tips and tricks The [implementation](/best-practices/how-we-mesh/mesh-4-implementation) page provides more in-depth examples of how to split a monolithic project into multiple projects. Here are some tips to get you started when considering the splitting methods listed above on your own projects: diff --git a/website/docs/best-practices/how-we-mesh/mesh-4-implementation.md b/website/docs/best-practices/how-we-mesh/mesh-4-implementation.md index 20910302a03..f1fb7422acf 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-4-implementation.md +++ b/website/docs/best-practices/how-we-mesh/mesh-4-implementation.md @@ -29,6 +29,9 @@ When attempting to define your project interfaces, you should consider investiga Let's go through an example process of taking a monolithing project, using groups and access to define the interfaces, and then splitting it into multiple projects. +To learn more, refer to our freely available [dbt Mesh learning course](https://learn.getdbt.com/courses/dbt-mesh). + + ## Defining project interfaces with groups and access Once you have a sense of some initial groupings, you can first implement **group and access permissions** within a single project. diff --git a/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md b/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md index 307d8ef1a98..f52cff78188 100644 --- a/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md +++ b/website/docs/best-practices/how-we-mesh/mesh-5-faqs.md @@ -78,7 +78,10 @@ dbt Mesh also enhances the interoperability and reusability of data across diffe -Like resource dependencies, project dependencies are acyclic, meaning they only move in one direction. This prevents `ref` cycles (or loops). For example, if project B depends on project A, a new model in project A could not import and use a public model from project B. Refer to [Project dependencies](/docs/collaborate/govern/project-dependencies#how-to-use-ref) for more information. +import CycleDetection from '/snippets/_mesh-cycle-detection.md'; + + + @@ -98,7 +101,7 @@ In the future, we plan for dbt Cloud to also be able to proactively flag this sc -No, unless downstream projects are installed as [packages](/docs/build/packages) (source code). In that case, the models in project installed as a project become “your” models, and you can select or run them. There are cases in which this can be desirable; see docs on [project dependencies](/docs/collaborate/govern/project-dependencies). +No, unless upstream projects are installed as [packages](/docs/build/packages) (source code). In that case, the models in project installed as a project become “your” models, and you can select or run them. There are cases in which this can be desirable; see docs on [project dependencies](/docs/collaborate/govern/project-dependencies). diff --git a/website/docs/community/resources/code-of-conduct.md b/website/docs/community/resources/code-of-conduct.md index 22159b36cc9..28185267bd4 100644 --- a/website/docs/community/resources/code-of-conduct.md +++ b/website/docs/community/resources/code-of-conduct.md @@ -102,6 +102,15 @@ Participants asked to stop any harassing behavior are expected to comply immedia If a participant engages in harassing behavior, the dbt Community Admin team may take any action they deem appropriate, up to and including expulsion from all dbt Community spaces and identification of the participant as a harasser to other dbt Community members or the general public. +## Coalesce and other events +We reserve the right to prohibit certain directly competitive companies from attending and / or sponsoring Coalesce and other dbt Labs-hosted events. + +While we actively support the broader ecosystem around dbt, it is not the right business decision for us to allow directly competitive companies to market to or sell to our users and customers at events that we invest very significant company resources into hosting. + +Any event declines will be handled directly with the individuals / companies in question, and full refunds will be issued. + +Events that are hosted by other members of the dbt community, such as most dbt Community Meetups, are free to make their own guidelines about attendance. + ## Credits Credit to [01.org](https://01.org/community/participation-guidelines), [Tizen.org](https://www.tizen.org/community/guidelines), and [Geek Feminism](https://geekfeminism.wikia.org/wiki/Community_anti-harassment/Policy) for some of the wording used in this Code of Conduct. diff --git a/website/docs/community/resources/oss-expectations.md b/website/docs/community/resources/oss-expectations.md index ba6c798e843..e6e5d959c96 100644 --- a/website/docs/community/resources/oss-expectations.md +++ b/website/docs/community/resources/oss-expectations.md @@ -107,3 +107,7 @@ PRs are your surest way to make the change you want to see in dbt / packages / d - **Tests!** When you open a PR, some tests and code checks will run. (For security reasons, some may need to be approved by a maintainer.) We will not merge any PRs with failing tests. If you’re not sure why a test is failing, please say so, and we’ll do our best to get to the bottom of it together. - **Contributor License Agreement** (CLA): This ensures that we can merge your code, without worrying about unexpected implications for the copyright or license of open source dbt software. For more details, read: ["Contributor License Agreements"](../resources/contributor-license-agreements.md) - **Changelog:** In projects that include a number of changes in each release, we need a reliable way to signal what's been included. The mechanism for this will vary by repository, so keep an eye out for notes about how to update the changelog. + +### Inclusion in release versions + +Both bug fixes and backwards-compatible new features will be included in the [next minor release](/docs/dbt-versions/core#how-dbt-core-uses-semantic-versioning). Fixes for regressions and net-new bugs that were present in the minor version's original release will be backported to versions with [active support](/docs/dbt-versions/core). Other bug fixes may be backported when we have high confidence that they're narrowly scoped and won't cause unintended side effects. diff --git a/website/docs/community/spotlight/meagan-palmer.md b/website/docs/community/spotlight/meagan-palmer.md new file mode 100644 index 00000000000..ff45a3d6b7d --- /dev/null +++ b/website/docs/community/spotlight/meagan-palmer.md @@ -0,0 +1,41 @@ +--- +id: meagan-palmer +title: Meagan Palmer +description: | + I first started using dbt in 2016 or 2017 (I can't remember exactly). Since then, I have moved into data and analytics consulting and have dipped in and out of the dbt Community. + Late last year, I started leading dbt Cloud training courses and spending more time in the dbt Slack. + In consulting, I get to use a range of stacks. I've used dbt with Redshift, Snowflake, and Databricks in production settings with a range of loaders & reporting tools, and I've been enjoying using DuckDB for some home experimentation. + To share some of the experiences, I regularly post to LinkedIn and have recently started Analytics Engineering Today, a twice monthly newsletter about dbt in practice. +image: /img/community/spotlight/Meagan-Palmer.png +pronouns: she/her +location: Sydney, Australia +jobTitle: Principal Consultant +companyName: Altis Consulting +socialLinks: + - name: LinkedIn + link: https://www.linkedin.com/in/meaganpalmer/ +dateCreated: 2024-07-29 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I was fortunate that Jon Bradley at Nearmap had the vision to engage the then Fishtown Analytics team (as the dbt Labs team was formerly called) as consultants and begin using dbt in our stack. I can't thank him enough. It was a turning point for my career, where I could combine my interests and experiences in delivering business value, data, product management, and software engineering practices. + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +Being in Australia, I often see replies from Jeremy Yeo to people in the dbt Slack. His clarity of communication is impressive. + +For growth, I'm hoping that others can benefit from the wide range of experience I have. My newsletter, Analytics Engineering Today on LinkedIn aims to upskill the dbt Community and shed some light on some useful features that might not be well known. + +I'll be at Coalesce and am doing some webinars/events later in the year. Come say hi, I love talking dbt and analytics engineering with people. + +## What have you learned from community members? What do you hope others can learn from you? + +The community members are amazing. It's great to be among a group of people that want to learn and improve. +I've learned a lot - both from other members helping with my queries and in reading how other businesses have implemented dbt, including their stories on the organizational & technical issues they face. + +I hope I can help instill a sense that simple, clean solutions are possible and preferable. I want to highlight that it is important to focus on what is the actual problem you are trying to solve and the fact that it's worth asking for help when you're starting to get stuck. + +I'm keen to help more women get the courage to work & lead in STEM. There has been a lot of progress made over the course of my career which is great to see. Australian/NZ women, please connect with me, happy to chat. diff --git a/website/docs/community/spotlight/mikko-sulonen.md b/website/docs/community/spotlight/mikko-sulonen.md new file mode 100644 index 00000000000..689b53db97e --- /dev/null +++ b/website/docs/community/spotlight/mikko-sulonen.md @@ -0,0 +1,33 @@ +--- +id: mikko-sulonen +title: Mikko Sulonen +description: | + I've been working with data since 2016. I first started with the on-prem SQL Server S-stack of SSIS, SSAS, SSRS. I did some QlikView and Qlik Sense, and some Power BI. Nowadays, I work mostly with Snowflake, Databricks, Azure, and dbt, of course. While tools and languages have come and gone, SQL has stayed. I've been a consultant for all of my professional life. +image: /img/community/spotlight/Mikko-Sulonen.png +pronouns: he/him +location: Tampere, Finland +jobTitle: Data Architect, Partner +companyName: Recordly Oy +socialLinks: + - name: LinkedIn + link: https://www.linkedin.com/in/mikkosulonen/ +dateCreated: 2024-07-28 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I started using dbt around 2019-2020, I think. Snapshots (no longer "archives") were a new thing along with the RPC server! I asked around my then-company: pretty much nobody had used dbt, though some commented that it did look promising. That left me looking elsewhere for experiences and best practices around the tool, and I found different blog writers and eventually the [dbt Slack](https://www.getdbt.com/community/join-the-community). I quickly noticed I could learn much more from the experiences of others than by trying everything myself. After just lurking for a while, I started to answer people's questions and give my own thoughts. This was completely new to me: voicing my input and opinions to people I had never met or who were not my colleagues. + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +There are quite many. I started to write some names here, but felt the list was getting a bit long and I'd still forget somebody! What the community leaders have in common is that they are approachable, knowledgeable, and passionate. They want to help others, they want to drive the community forward, and they are down to earth. I've had the pleasure of meeting many of them in person at the past two [Coalesces](https://coalesce.getdbt.com/), and I hope to meet many more! + +Growing my own leadership in the community... That's an interesting question: I hadn't really identified myself as leader in the community before. Maybe I should come out of the Slack channels and join and/or organize some [dbt Meetups](https://www.meetup.com/pro/dbt/)? I always try to answer even the simplest questions, even if they've been answered a hundred times already. Every day, new people are introduced to dbt, and they are facing issues for the first time. Every one of us was new at one point! + +## What have you learned from community members? What do you hope others can learn from you? + +I've learnt a surprising amount about the different teams and ways of working with and around data. I've learnt that it is highly probable that somebody somewhere has already had, and likely solved, the problem you are having. All that is needed to connect the dots is for people to speak and listen. + +When asking and answering questions, I try to hone in on what they're really trying to get at. I ask, why is it that you want to do something in that particular way? I like to say, don't love the solutions, love the problems! diff --git a/website/docs/community/spotlight/radovan-bacovic.md b/website/docs/community/spotlight/radovan-bacovic.md new file mode 100644 index 00000000000..186edccac6d --- /dev/null +++ b/website/docs/community/spotlight/radovan-bacovic.md @@ -0,0 +1,42 @@ +--- +id: radovan-bacovic +title: Radovan Bacovic +description: | + My professional journey and friendship with data started 20 years ago. + I've experienced many tools and modalities: from good old RDMS systems and various programming languages like Java and C# in the early days of my career, through private clouds, to MPP databases and multitier architecture. I also saw the emergence of cloud technology, and experienced the changes that came with it, up to the contemporary approach to data, which includes AI tools and practices. I am still excited to get new knowledge and solve problems in the data world. I always enjoy using SQL and Python as my primary "weapons" of choice together with other building blocks for successful data products like Snowflake, dbt, Tableau, Fivetran, Stich, Monte Carlo, DuckDB and more. + I consider myself as an experienced data engineer and a wannabe "best bad Conference speaker." +image: /img/community/spotlight/Radovan-Bacovic.png +pronouns: he/him +location: Novi Sad, Serbia +jobTitle: Staff Data Engineer +companyName: GitLab +organization: Permifrost maintainer +socialLinks: + - name: X + link: https://x.com/svenmurtinson + - name: LinkedIn + link: https://www.linkedin.com/in/radovan-ba%C4%87ovi%C4%87-6498603/ + - name: Gitlab Profile + link: https://gitlab.com/rbacovic +dateCreated: 2024-07-28 +hide_table_of_contents: true +communityAward: false +--- + +## When did you join the dbt community and in what way has it impacted your career? + +I have been in the dbt Community for almost 3 years now. The biggest impact that dbt has had on my professional journey is that it has given me a trustworthy partner for data transformation, and a single source of truth for all our data modification needs. As a public speaker who travels internationally, I recognized the interest of the data community in dbt around the world and, in response, organised several workshops and talks to help people use dbt. Let's just say that jumping into a great partnership with the dbt Community has been the greatest takeaway for me! + +## Which dbt community leader do you identify with? How are you looking to grow your leadership in the dbt community? + +A leader from the dbt Commuity who I have found to be the most prominent is Sean McIntyre from dbt Labs, as I've had the privilege to collaborate with him many times. We recognized that we had a similar passion and energy; it looks like we are on the same journey. + +I wanted to be more involved in the dbt Community, spread the word, and share my journey through tutorials, conference talks, blogs and meetups. I think I am capable of addressing my influence in that direction. I am also interested in extending the dbt functionality and automating the deployment, testing and execution of dbt. In other words, I try to find good ways to leverage using DevSecOps for dbt to make our development faster and make dbt our trustworthy partner. + +## What have you learned from community members? What do you hope others can learn from you? + +Being part of the community is always a two-way street and always has a positive result for all of us. Great ideas and sharing vision and energy are the number one things. On the other side, I always find the quick and "best in class" answer to my questions from the community and try to return the favour and be helpful whenever possible. As I said, through talks and tutorials, I think I am the most beneficial community member in that role. + +## Anything else interesting you want to tell us? + +Nothing to add. Still passionate about discovering the tool and can't wait to see what the AI uprising will bring to us! diff --git a/website/docs/docs/build/custom-aliases.md b/website/docs/docs/build/custom-aliases.md index 9e9f91f968d..ee56480dc5c 100644 --- a/website/docs/docs/build/custom-aliases.md +++ b/website/docs/docs/build/custom-aliases.md @@ -73,8 +73,6 @@ To override dbt's alias name generation, create a macro named `generate_alias_na The default implementation of `generate_alias_name` simply uses the supplied `alias` config (if present) as the model alias, otherwise falling back to the model name. This implementation looks like this: - - ```jinja2 @@ -100,8 +98,6 @@ The default implementation of `generate_alias_name` simply uses the supplied `al - - import WhitespaceControl from '/snippets/_whitespace-control.md'; diff --git a/website/docs/docs/build/custom-databases.md b/website/docs/docs/build/custom-databases.md index 3ffd8ae3354..7a607534230 100644 --- a/website/docs/docs/build/custom-databases.md +++ b/website/docs/docs/build/custom-databases.md @@ -6,7 +6,7 @@ id: "custom-databases" :::info A word on naming -Different warehouses have different names for "logical databases". The information in this document covers "databases" on Snowflake, Redshift, and Postgres, as well as "projects" on BigQuery. +Different warehouses have different names for _logical databases_. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. The values `project` and `database` are interchangeable in BigQuery project configurations. @@ -89,14 +89,10 @@ import WhitespaceControl from '/snippets/_whitespace-control.md'; - - ### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) - - ## Considerations ### BigQuery diff --git a/website/docs/docs/build/custom-schemas.md b/website/docs/docs/build/custom-schemas.md index 846f2d1c341..6dabf56943c 100644 --- a/website/docs/docs/build/custom-schemas.md +++ b/website/docs/docs/build/custom-schemas.md @@ -154,14 +154,10 @@ The following context methods _are_ available in the `generate_schema_name` macr Globally-scoped variables and variables defined on the command line with [--vars](/docs/build/project-variables) are accessible in the `generate_schema_name` context. - - ### Managing different behaviors across packages See docs on macro `dispatch`: ["Managing different global overrides across packages"](/reference/dbt-jinja-functions/dispatch) - - ## A built-in alternative pattern for generating schema names A common customization is to ignore the target schema in production environments, and ignore the custom schema configurations in other environments (such as development and CI). diff --git a/website/docs/docs/build/data-tests.md b/website/docs/docs/build/data-tests.md index 38a2c74b646..158b8427136 100644 --- a/website/docs/docs/build/data-tests.md +++ b/website/docs/docs/build/data-tests.md @@ -19,7 +19,7 @@ keywords: :::important -In dbt v1.8, what was previously known as "tests" are now called "data tests" with the addition of [unit tests](/docs/build/unit-tests). The YAML key `tests:` is still supported as an alias for data tests but will be deprecated in the future in favor of `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. +From dbt v1.8, "tests" are now called "data tests" to disambiguate from [unit tests](/docs/build/unit-tests). The YAML key `tests:` is still supported as an alias for `data_tests:`. Refer to [New `data_tests:` syntax](#new-data_tests-syntax) for more information. ::: @@ -275,13 +275,11 @@ In dbt version 1.8, we updated the `tests` configuration to `data_tests`. For de - + -Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, it was necessary to update our naming conventions and syntax. - -As of v1.8, `tests:` is still supported in your YML configuration files as an alias but will be deprecated in the future in favor of `data_tests:`. +Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, the key was renamed from `tests:` to `data_tests:`. -As we progress towards this deprecation, we'll update the examples in our docs pages to reflect this new syntax, but we highly recommend you begin the migration process as soon as you upgrade to v1.8 to avoid interruptions or issues in the future. +dbt still supports `tests:` in your YML configuration files for backwards-compatibility purposes, and you might see it used throughout our documentation. However, you can't have a `tests` and a `data_tests` key associated with the same resource (e.g. a single model) at the same time. @@ -306,6 +304,8 @@ data_tests: +To suppress warnings about the rename, add `TestsConfigDeprecation` to the `silence` block of the `warn_error_options` flag in `dbt_project.yml`, [as described in the Warnings documentation](https://docs.getdbt.com/reference/global-configs/warnings). + ## FAQs diff --git a/website/docs/docs/build/documentation.md b/website/docs/docs/build/documentation.md index e57e7fea679..d040d3c5bef 100644 --- a/website/docs/docs/build/documentation.md +++ b/website/docs/docs/build/documentation.md @@ -59,12 +59,12 @@ models: The default documentation experience in dbt Cloud is [dbt Explorer](/docs/collaborate/explore-projects), available on [Team or Enterprise plans](https://www.getdbt.com/pricing/). Use dbt Explorer to view your project's resources (such as models, tests, and metrics), its [metadata](/docs/collaborate/explore-projects#generate-metadata), and their lineage to gain a better understanding of its latest production state. -dbt Cloud developer and dbt Core users can use [dbt Docs](/docs/collaborate/build-and-view-your-docs#dbt-docs), which generates basic documentation, but it doesn't offer the same speed, metadata, or visibility as dbt Explorer. +dbt Cloud developer and dbt Core users can use [dbt Docs](/docs/collaborate/build-and-view-your-docs#dbt-docs), which generates basic documentation, but doesn't offer the same speed, metadata, or visibility as dbt Explorer. -Generate documentation for you project by following these steps: +Generate documentation for your project by following these steps: 1. Run `dbt docs generate` — this command tells dbt to compile relevant information about your dbt project and warehouse into `manifest.json` and `catalog.json` files, respectively. -2. Ensure that you have created the models with `dbt run` to view the documentation for all columns, not just those described in your project. +2. Ensure you've created the models with `dbt run` to view the documentation for all columns, not just those described in your project. 3. Run `dbt docs serve` if you're developing locally to use these `.json` files to populate a local website. To view a resource, its metadata, and what commands are needed in dbt Explorer, refer to [generate metadata](/docs/collaborate/explore-projects#generate-metadata) for more details. diff --git a/website/docs/docs/build/environment-variables.md b/website/docs/docs/build/environment-variables.md index d009cb9663a..01601ce7eb8 100644 --- a/website/docs/docs/build/environment-variables.md +++ b/website/docs/docs/build/environment-variables.md @@ -8,7 +8,9 @@ Environment variables can be used to customize the behavior of a dbt project dep [env_var](/reference/dbt-jinja-functions/env_var) for more information on how to call the jinja function `{{env_var('DBT_KEY','OPTIONAL_DEFAULT')}}` in your project code. :::info Environment Variable Naming and Prefixing -Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. + +Environment variables in dbt Cloud must be prefixed with either `DBT_` or `DBT_ENV_SECRET` or `DBT_ENV_CUSTOM_ENV_`. Environment variables keys are uppercased and case sensitive. When referencing `{{env_var('DBT_KEY')}}` in your project's code, the key must match exactly the variable defined in dbt Cloud's UI. + ::: ### Setting and overriding environment variables @@ -20,11 +22,11 @@ Environment variable values can be set in multiple places within dbt Cloud. As a There are four levels of environment variables: - 1. the optional default argument supplied to the `env_var` Jinja function in code - 2. a project-wide default value, which can be overridden at - 3. the environment level, which can in turn be overridden again at - 4. the job level (job override) or in the IDE for an individual dev (personal override). + 1. The optional default argument supplied to the `env_var` Jinja function in code, which can be overridden at (_lowest precedence_) + 2. The project-wide level by its default value, which can be overridden at + 3. The environment level, which can in turn be overridden again at + 4. The job level (job override) or in the IDE for an individual dev (personal override). (_highest precedence_) **Setting environment variables at the project and environment level** @@ -36,11 +38,12 @@ To set environment variables at the project and environment level, click **Deplo You'll notice there is a `Project Default` column. This is a great place to set a value that will persist across your whole project, independent of where the code is run. We recommend setting this value when you want to supply a catch-all default or add a project-wide token or secret. -To the right of the `Project Default` column are all your environments. Values set at the environment level take priority over the project-level default value. This is where you can tell dbt Cloud to interpret an environment value differently in your Staging vs. Production environment, as an example. - +To the right of the `Project Default` column are all your environments. Values set at the environmental level take priority over the project-level default value. This is where you can tell dbt Cloud to interpret an environment value differently in your Staging vs. Production environment, as an example. + + **Overriding environment variables at the job level** @@ -59,7 +62,7 @@ Every job runs in a specific, deployment environment, and by default, a job will **Overriding environment variables at the personal level** -You can also set a personal value override for an environment variable when you develop in the dbt integrated developer environment (IDE). By default, dbt Cloud uses environment variable values set in the project's development environment. To see and override these values, click the gear icon in the top right. Under "Your Profile," click **Credentials** and select your project. Click **Edit** and make any changes in "Environment Variables." +You can also set a personal value override for an environment variable when you develop in the dbt-integrated developer environment (IDE). By default, dbt Cloud uses environment variable values set in the project's development environment. To see and override these values, click the gear icon in the top right. Under "Your Profile," click **Credentials** and select your project. Click **Edit** and make any changes in "Environment Variables." @@ -83,7 +86,7 @@ There are some known issues with partial parsing of a project and changing envir ### Handling secrets -While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in the UI, you can prefix the key with `DBT_ENV_SECRET_``DBT_ENV_SECRET`. This functionality is supported from `dbt v1.0` and on. +While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud has additional capabilities for managing environment variables with secret or otherwise sensitive values. If you want a particular environment variable to be scrubbed from all logs and error messages, in addition to obfuscating the value in the UI, you can prefix the key with `DBT_ENV_SECRET`. This functionality is supported from `dbt v1.0` and on. @@ -94,8 +97,6 @@ While all environment variables are encrypted at rest in dbt Cloud, dbt Cloud ha dbt Cloud has a number of pre-defined variables built in. Variables are set automatically and cannot be changed. - - **dbt Cloud IDE details** The following environment variable is set automatically for the dbt Cloud IDE: @@ -108,8 +109,6 @@ The following environment variable is set automatically for the dbt Cloud IDE: Use case — This is useful in cases where you want to dynamically use the Git branch name as a prefix for a [development schema](/docs/build/custom-schemas) ( `{{ env_var ('DBT_CLOUD_GIT_BRANCH') }}` ). - - **dbt Cloud context** The following environment variables are set automatically for deployment runs: diff --git a/website/docs/docs/build/incremental-models.md b/website/docs/docs/build/incremental-models.md index 21215437d89..21cd656484a 100644 --- a/website/docs/docs/build/incremental-models.md +++ b/website/docs/docs/build/incremental-models.md @@ -71,7 +71,7 @@ from {{ ref('app_data_events') }} -- this filter will only be applied on an incremental run -- (uses >= to include records whose timestamp occurred since the last run of this model) -- (If event_time is NULL or the table is truncated, the condition will always be true and load all records) -where event_time >= (select coalesce(max(event_time),'1900-01-01'::TIMESTAMP) from {{ this }} ) +where event_time >= (select coalesce(max(event_time),'1900-01-01') from {{ this }} ) {% endif %} ``` diff --git a/website/docs/docs/build/incremental-strategy.md b/website/docs/docs/build/incremental-strategy.md index 09a9187f8aa..8e86da0eba8 100644 --- a/website/docs/docs/build/incremental-strategy.md +++ b/website/docs/docs/build/incremental-strategy.md @@ -19,22 +19,6 @@ Click the name of the adapter in the below table for more information about supp The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in dbt v1.6. - - -| data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | -|-----------------------------------------------------------------------------------------------------|:--------:|:-------:|:---------------:|:------------------:| -| [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | ✅ | | ✅ | | -| [dbt-redshift](/reference/resource-configs/redshift-configs#incremental-materialization-strategies) | ✅ | | ✅ | | -| [dbt-bigquery](/reference/resource-configs/bigquery-configs#merge-behavior-incremental-models) | | ✅ | | ✅ | -| [dbt-spark](/reference/resource-configs/spark-configs#incremental-models) | ✅ | ✅ | | ✅ | -| [dbt-databricks](/reference/resource-configs/databricks-configs#incremental-models) | ✅ | ✅ | | ✅ | -| [dbt-snowflake](/reference/resource-configs/snowflake-configs#merge-behavior-incremental-models) | ✅ | ✅ | ✅ | | -| [dbt-trino](/reference/resource-configs/trino-configs#incremental) | ✅ | ✅ | ✅ | | - - - - - | data platform adapter | `append` | `merge` | `delete+insert` | `insert_overwrite` | |-----------------------------------------------------------------------------------------------------|:--------:|:-------:|:---------------:|:------------------:| | [dbt-postgres](/reference/resource-configs/postgres-configs#incremental-materialization-strategies) | ✅ | ✅ | ✅ | | @@ -46,7 +30,6 @@ The `merge` strategy is available in dbt-postgres and dbt-redshift beginning in | [dbt-trino](/reference/resource-configs/trino-configs#incremental) | ✅ | ✅ | ✅ | | | [dbt-fabric](/reference/resource-configs/fabric-configs#incremental) | ✅ | | ✅ | | - :::note Snowflake Configurations diff --git a/website/docs/docs/build/join-logic.md b/website/docs/docs/build/join-logic.md index cf4c1608284..99d63b38657 100644 --- a/website/docs/docs/build/join-logic.md +++ b/website/docs/docs/build/join-logic.md @@ -122,7 +122,7 @@ semantic_models: - name: user_id type: primary - name: country_id - type: Unique + type: unique dimensions: - name: signup_date type: time diff --git a/website/docs/docs/build/measures.md b/website/docs/docs/build/measures.md index 5e0772f517e..9458487e8d4 100644 --- a/website/docs/docs/build/measures.md +++ b/website/docs/docs/build/measures.md @@ -141,13 +141,13 @@ semantic_models: description: Distinct count of transactions expr: transaction_id agg: count_distinct - - name: transactions + - name: transaction_amount_avg description: The average value of transactions expr: transaction_amount_usd agg: average - name: transactions_amount_usd_valid # Notice here how we use expr to compute the aggregation based on a condition description: The total USD value of valid transactions only - expr: CASE WHEN is_valid = True then 1 else 0 end + expr: CASE WHEN is_valid = True then transaction_amount_usd else 0 end agg: sum - name: transactions description: The average value of transactions. diff --git a/website/docs/docs/build/metricflow-commands.md b/website/docs/docs/build/metricflow-commands.md index 09822fd70ad..c8ba951733c 100644 --- a/website/docs/docs/build/metricflow-commands.md +++ b/website/docs/docs/build/metricflow-commands.md @@ -67,7 +67,7 @@ MetricFlow provides the following commands to retrieve metadata and query metric -You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. +You can use the `dbt sl` prefix before the command name to execute them in the dbt Cloud CLI. For example, to list all metrics, run `dbt sl list metrics`. For a complete list of the MetricFlow commands and flags, run the `dbt sl --help` command in your terminal. - [`list`](#list) — Retrieves metadata values. - [`list metrics`](#list-metrics) — Lists metrics with dimensions. @@ -76,6 +76,9 @@ You can use the `dbt sl` prefix before the command name to execute them in the d - [`list entities`](#list-entities) — Lists all unique entities. - [`list saved queries`](#list-saved-queries) — Lists available saved queries. Use the `--show-exports` flag to display each export listed under a saved query. - [`query`](#query) — Query metrics, saved queries, and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started. +- [`export`](#export) — Runs exports for a singular saved query for testing and generating exports in your development environment. You can also use the `--select` flag to specify particular exports from a saved query. +- [`export-all`](#export-all) — Runs exports for multiple saved queries at once, saving time and effort. + - - ### Which metrics are available? -You can define and query metrics using the [dbt Semantic Layer](/docs/build/about-metricflow), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL). To learn more, refer to [Get started with MetricFlow](/docs/build/sl-getting-started). +You can define and query metrics using the [dbt Semantic Layer](/docs/build/about-metricflow), use them for documentation purposes (like for a data catalog), and calculate aggregations (like in a BI tool that doesn’t query the SL).
Example query @@ -944,10 +938,6 @@ query ($environmentId: BigInt!, $first: Int!) {
-
- - - ## Governance You can use the Discovery API to audit data development and facilitate collaboration within and between teams. @@ -1041,8 +1031,6 @@ query ($environmentId: BigInt!, $first: Int!) { ``` - - ## Development You can use the Discovery API to understand dataset changes and usage and gauge impacts to inform project definition. Below are example questions and queries you can run. diff --git a/website/docs/docs/dbt-cloud-apis/schema-discovery-environment-applied-modelHistoricalRuns.mdx b/website/docs/docs/dbt-cloud-apis/schema-discovery-environment-applied-modelHistoricalRuns.mdx index d1463f9e9b7..e76a7130c80 100644 --- a/website/docs/docs/dbt-cloud-apis/schema-discovery-environment-applied-modelHistoricalRuns.mdx +++ b/website/docs/docs/dbt-cloud-apis/schema-discovery-environment-applied-modelHistoricalRuns.mdx @@ -25,7 +25,7 @@ query { environment(id: 834) { applied { modelHistoricalRuns( - uniqueId: "model.marketing.customers" + uniqueId: "model.marketing.customers" # Use this format for unique ID: RESOURCE_TYPE.PACKAGE_NAME.RESOURCE_NAME lastRunCount: 20 ) { runId # Get historical results for a particular model diff --git a/website/docs/docs/dbt-cloud-apis/service-tokens.md b/website/docs/docs/dbt-cloud-apis/service-tokens.md index 26b0e58caa2..1a5920fab8a 100644 --- a/website/docs/docs/dbt-cloud-apis/service-tokens.md +++ b/website/docs/docs/dbt-cloud-apis/service-tokens.md @@ -69,6 +69,9 @@ Security Admin service tokens have certain account-level permissions. For more **Billing Admin**
Billing Admin service tokens have certain account-level permissions. For more on these permissions, see [Billing Admin](/docs/cloud/manage-access/enterprise-permissions#billing-admin). +**Manage marketplace apps**
+Used only for service tokens assigned to marketplace apps (for example, the [Snowflake Native app](/docs/cloud-integrations/snowflake-native-app)). + **Metadata Only**
Metadata-only service tokens authorize requests to the Discovery API. diff --git a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md index 9c3bf912597..1c4d5f387e9 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-api-overview.md +++ b/website/docs/docs/dbt-cloud-apis/sl-api-overview.md @@ -6,14 +6,6 @@ tags: [Semantic Layer, API] hide_table_of_contents: true pagination_next: "docs/dbt-cloud-apis/sl-jdbc" --- - - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - The rapid growth of different tools in the modern data stack has helped data professionals address the diverse needs of different teams. The downside of this growth is the fragmentation of business logic across teams, tools, and workloads.

@@ -50,6 +42,12 @@ plan="dbt Cloud Team or Enterprise" link="/docs/dbt-cloud-apis/sl-graphql" icon="dbt-bit"/> + + - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - -
- - [GraphQL](https://graphql.org/) (GQL) is an open-source query language for APIs. It offers a more efficient and flexible approach compared to traditional RESTful APIs. With GraphQL, users can request specific data using a single query, reducing the need for many server round trips. This improves performance and minimizes network overhead. diff --git a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md index bb5c8fe7918..672f34b2fec 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-jdbc.md +++ b/website/docs/docs/dbt-cloud-apis/sl-jdbc.md @@ -5,14 +5,6 @@ description: "Integrate and use the JDBC API to query your metrics." tags: [Semantic Layer, API] --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer Java Database Connectivity (JDBC) API enables users to query metrics and dimensions using the JDBC protocol, while also providing standard metadata functionality. A JDBC driver is a software component enabling a Java application to interact with a data platform. Here's some more information about our JDBC API: diff --git a/website/docs/docs/dbt-cloud-apis/sl-manifest.md b/website/docs/docs/dbt-cloud-apis/sl-manifest.md index 0dbbeb7b695..e203f4a0754 100644 --- a/website/docs/docs/dbt-cloud-apis/sl-manifest.md +++ b/website/docs/docs/dbt-cloud-apis/sl-manifest.md @@ -7,14 +7,6 @@ sidebar_label: "Semantic manifest" pagination_next: null --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - dbt creates an [artifact](/reference/artifacts/dbt-artifacts) file called the _Semantic Manifest_ (`semantic_manifest.json`), which MetricFlow requires to build and run metric queries properly for the dbt Semantic Layer. This artifact contains comprehensive information about your dbt Semantic Layer. It is an internal file that acts as the integration point with MetricFlow. By using the semantic manifest produced by dbt Core, MetricFlow will instantiate a data flow plan and generate SQL from Semantic Layer query requests. It's a valuable reference that you can use to understand the structure and details of your data models. diff --git a/website/docs/docs/dbt-cloud-apis/sl-python-sdk.md b/website/docs/docs/dbt-cloud-apis/sl-python-sdk.md new file mode 100644 index 00000000000..94364687822 --- /dev/null +++ b/website/docs/docs/dbt-cloud-apis/sl-python-sdk.md @@ -0,0 +1,152 @@ +--- +title: "Python SDK" +id: sl-python +description: "Learn how to use the dbt Semantic Layer Python SDK library to interact with the dbt Semantic Layer." +tags: [Semantic Layer, APIs] +keywords: [dbt Cloud, API, dbt Semantic Layer, python, sdk] +sidebar_label: "Python SDK" +--- + +# Python SDK +The [`dbt-sl-sdk` Python software development kit](https://github.com/dbt-labs/semantic-layer-sdk-python) (SDK) is a Python library that provides you with easy access to the dbt Semantic Layer with Python. It allows developers to interact with the dbt Semantic Layer APIs and query metrics and dimensions in downstream tools. + +## Installation + +To install the Python SDK, you'll need to specify optional dependencies depending on whether you want to use it synchronously, backed by [requests](https://github.com/psf/requests/), or with asynchronous ([asyncio](https://docs.python.org/3/library/asyncio.html) backed by [aiohttp](https://github.com/aio-libs/aiohttp/)). + +The Python SDK supports the Long-Term Support (LTS) versions of Python, such as 3.9, 3.10, 3.11, and 3.12. When Python discontinues support for a version, the Python SDK will also discontinue support for that version. If you’re using a non-supported version, you may experience compatibility issues and won’t receive updates or security patches from the SDK. + + + + +Sync installation means your program waits for each task to finish before moving on to the next one. + +It's simpler, easier to understand, and suitable for smaller tasks or when your program doesn't need to handle many tasks at the same time. + +```bash +pip install dbt-sl-sdk[sync] +``` +If you're using async frameworks like [FastAPI](https://fastapi.tiangolo.com/) or [Strawberry](https://github.com/strawberry-graphql/strawberry), installing the sync version of the SDK will block your event loop and can significantly slow down your program. In this case, we strongly recommend using async installation. + + + + + +Async installation means your program can start a task and then move on to other tasks while waiting for the first one to finish. This can handle many tasks at once without waiting, making it faster and more efficient for larger tasks or when you need to manage multiple tasks at the same time. + +For more details, refer to [asyncio](https://docs.python.org/3/library/asyncio.html). + +```bash +pip install dbt-sl-sdk[async] +``` + +Since the [Python ADBC driver](https://github.com/apache/arrow-adbc/tree/main/python/adbc_driver_manager) doesn't yet support asyncio natively, `dbt-sl-sdk` uses a [`ThreadPoolExecutor`](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/5e52e1ca840d20a143b226ae33d194a4a9bc008f/dbtsl/api/adbc/client/asyncio.py#L62) to run `query` and `list dimension-values` (all operations that are done with ADBC). This is why you might see multiple Python threads spawning. + +If you're using async frameworks like [FastAPI](https://fastapi.tiangolo.com/) or [Strawberry](https://github.com/strawberry-graphql/strawberry), installing the sync version of the Python SDK will block your event loop and can significantly slow down your program. In this case, we strongly recommend using async installation. + + + + +## Usage +To run operations against the Semantic Layer APIs, instantiate (create an instance of) a `SemanticLayerClient` with your specific [API connection parameters](/docs/dbt-cloud-apis/sl-api-overview): + +```python +from dbtsl import SemanticLayerClient + +client = SemanticLayerClient( + environment_id=123, + auth_token="", + host="semantic-layer.cloud.getdbt.com", +) + +# query the first metric by `metric_time` +def main(): + with client.session(): + metrics = client.metrics() + table = client.query( + metrics=[metrics[0].name], + group_by=["metric_time"], + ) + print(table) + +main() +``` + +**Note**: All method calls that reach out to the APIs need to be within a `client.session()` context manager. This allows the client to establish a connection to the APIs only once and reuse the same connection between API calls. + +We recommend creating an application-wide session and reusing the same session throughout the application for optimal performance. Creating a session per request is discouraged and inefficient. + +### asyncio usage +If you're using asyncio, import `AsyncSemanticLayerClient` from `dbtsl.asyncio`. The `SemanticLayerClient` and `AsyncSemanticLayerClient` APIs are identical, but the async version has async methods that you need to `await`. + +```python +import asyncio +from dbtsl.asyncio import AsyncSemanticLayerClient + +client = AsyncSemanticLayerClient( + environment_id=123, + auth_token="", + host="semantic-layer.cloud.getdbt.com", +) + +async def main(): + async with client.session(): + metrics = await client.metrics() + table = await client.query( + metrics=[metrics[0].name], + group_by=["metric_time"], + ) + print(table) + +asyncio.run(main()) + +``` + +## Integrate with dataframe libraries + +The Python SDK returns all query data as [pyarrow](https://arrow.apache.org/docs/python/index.html) tables. + +The Python SDK library doesn't come bundled with [Polars](https://pola.rs/) or [Pandas](https://pandas.pydata.org/). If you use these libraries, add them as dependencies in your project. + +To use the data with libraries like Polars or Pandas, manually convert the data into the desired format. For example: + +#### If you're using pandas + +```python +# ... initialize client + +arrow_table = client.query(...) +pandas_df = arrow_table.to_pandas() + +``` + +#### If you're using polars + +```python +import polars as pl + +# ... initialize client + +arrow_table = client.query(...) +polars_df = pl.from_arrow(arrow_table) +``` + +## Usage examples +For additional usage examples, check out the [usage examples](https://github.com/dbt-labs/semantic-layer-sdk-python/tree/main/examples), some of which include: + +- [Fetching dimension values sync](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_dimension_values_sync.py) +- Fetching metrics [async](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_metric_async.py) and [sync](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/fetch_metric_sync.py) +- [List saved queries async](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/examples/list_saved_queries_async.py) + +## Disable telemetry +By default, the Python SDK sends some [platform-related information](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/dbtsl/env.py) to dbt Labs. To opt-out, set the `PLATFORM.anonymous` attribute to `True`: + +```python +from dbtsl.env import PLATFORM +PLATFORM.anonymous = True + +# ... initialize client +``` + +## Contribute +To contribute to this project, check out our [contribution guidelines](https://github.com/dbt-labs/semantic-layer-sdk-python/blob/main/CONTRIBUTING.md) and open a GitHub [issue](https://github.com/dbt-labs/semantic-layer-sdk-python/issues) or [pull request](https://github.com/dbt-labs/semantic-layer-sdk-python/pulls). diff --git a/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md b/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md index e5d35f6b597..544590b18df 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md +++ b/website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.9.md @@ -9,7 +9,7 @@ displayed_sidebar: "docs" - Changelog (coming soon) - [dbt Core CLI Installation guide](/docs/core/installation-overview) -- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud) — dbt Cloud is now versionless. dbt v1.9 will not appear in the version dropdown. Select "Keep on latest version" to get all the latest features and functionality in your dbt Cloud account. +- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud) — dbt Cloud is now versionless. dbt v1.9 will not appear in the version dropdown. Select **Versionless** to get all the latest features and functionality in your dbt Cloud account. ## What to know before upgrading diff --git a/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md b/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md index e1e57b2e3a4..dd22329668c 100644 --- a/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md +++ b/website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.8.md @@ -15,11 +15,11 @@ displayed_sidebar: "docs" dbt Labs is committed to providing backward compatibility for all versions 1.x, except for any changes explicitly mentioned on this page. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new). -## Keep on latest version +## Versionless dbt Cloud is going "versionless." This means you'll automatically get early access to new features and functionality before they're available in final releases of dbt Core. -Select ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) in your development, staging, and production [environments](/docs/deploy/deploy-environments) to access to everything in dbt Core v1.8 and more. +Select [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) in your development, staging, and production [environments](/docs/deploy/deploy-environments) to access to everything in dbt Core v1.8+ and more. To upgrade an environment in the [dbt Cloud Admin API](/docs/dbt-cloud-apis/admin-cloud-api) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest), set `dbt_version` to the string `versionless`. diff --git a/website/docs/docs/dbt-versions/core-versions.md b/website/docs/docs/dbt-versions/core-versions.md index f94a4c0cdf3..4a490f96bd5 100644 --- a/website/docs/docs/dbt-versions/core-versions.md +++ b/website/docs/docs/dbt-versions/core-versions.md @@ -12,7 +12,7 @@ dbt Core releases follow [semantic versioning](https://semver.org/) guidelines. _Did you know that you can always be working with the latest features and functionality?_ -With dbt Cloud, you can get early access to new functionality before it becomes available in dbt Core and without the need of managing your own version upgrades. Refer to the [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) setting for details. +With dbt Cloud, you can get early access to new functionality before it becomes available in dbt Core and without the need of managing your own version upgrades. Refer to the [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) setting for details. ::: diff --git a/website/docs/docs/dbt-versions/product-lifecycles.md b/website/docs/docs/dbt-versions/product-lifecycles.md index 313be55228a..e8711c825c4 100644 --- a/website/docs/docs/dbt-versions/product-lifecycles.md +++ b/website/docs/docs/dbt-versions/product-lifecycles.md @@ -17,7 +17,7 @@ dbt Cloud features all fall into one of the following categories: - **Beta:** Beta features are still in development and are only available to select customers. To join a beta, there might be a signup form or dbt Labs may contact specific customers about testing. Some features can be activated by enabling [experimental features](/docs/dbt-versions/experimental-features) in your account. Beta features are incomplete and might not be entirely stable; they should be used at the customer’s risk, as breaking changes could occur. Beta features might not be fully documented, technical support is limited, and service level objectives (SLOs) might not be provided. Download the [Beta Features Terms and Conditions](/assets/beta-tc.pdf) for more details. - **Preview:** Preview features are stable and considered functionally ready for production deployments. Some planned additions and modifications to feature behaviors could occur before they become generally available. New functionality that is not backward compatible could also be introduced. Preview features include documentation, technical support, and service level objectives (SLOs). Features in preview are provided at no extra cost, although they might become paid features when they become generally available. -- **Generally available (GA):** Generally available features provide stable features introduced to all qualified dbt Cloud accounts. Service level agreements (SLAs) apply to GA features, including documentation and technical support. Certain GA feature availability is determined by the dbt version of the environment. To always receive the latest GA features, go versionless and ensure your dbt Cloud [environments](/docs/dbt-cloud-environments) are set to ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). +- **Generally available (GA):** Generally available features provide stable features introduced to all qualified dbt Cloud accounts. Service level agreements (SLAs) apply to GA features, including documentation and technical support. Certain GA feature availability is determined by the dbt version of the environment. To always receive the latest GA features, ensure your dbt Cloud [environments](/docs/dbt-cloud-environments) are set to ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless). - **Deprecated:** Features in this state are no longer being developed or enhanced by dbt Labs. They will continue functioning as-is, and their documentation will persist until their removal date. However, they are no longer subject to technical support. - **Removed:** Removed features are no longer available on the platform in any capacity. diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 5ce0c210b95..b48dd344dfc 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -16,11 +16,30 @@ dbt Cloud release notes for recent and historical changes. Release notes fall in Release notes are grouped by month for both multi-tenant and virtual private cloud (VPC)[^*] environments -[^*] The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. +[^*] The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## July 2024 +- **New:** [Connections](/docs/cloud/connect-data-platform/about-connections#connection-management) are now available under **Account settings** as a global setting. Previously, they were found under **Project settings**. This is being rolled out in phases over the coming weeks. +- **New:** Admins can now assign [environment-level permissions](/docs/cloud/manage-access/environment-permissions) to groups for specific roles. +- **New:** [Merge jobs](/docs/deploy/merge-jobs) for implementing [continuous deployment (CD)](/docs/deploy/continuous-deployment) workflows are now GA in dbt Cloud. Previously, you had to either set up a custom GitHub action or manually build the changes every time a pull request is merged. - **New**: The ability to lint your SQL files from the dbt Cloud CLI is now available. To learn more, refer to [Lint SQL files](/docs/cloud/configure-cloud-cli#lint-sql-files). +- **Behavior change:** dbt Cloud IDE automatically adds a `--limit 100` to preview queries to avoid slow and expensive queries during development. Recently, dbt Core changed how the `limit` is applied to ensure that `order by` clauses are consistently respected. Because of this, queries that already contain a limit clause might now cause errors in the IDE previews. To address this, dbt Labs plans to provide an option soon to disable the limit from being applied. Until then, dbt Labs recommends removing the (duplicate) limit clause from your queries during previews to avoid these IDE errors. +- **Enhancement:** Custom configurations are now supported by generic data tests. Use this to configure how dbt should run the data test (for example, specifying a Snowflake virtual warehouse different from the one in your connection). To learn more, refer to [Specify custom configurations for generic data tests](/reference/data-test-configs#specify-custom-configurations-for-generic-data-tests). + + Support for this configuration is available in dbt Cloud when selecting **Versionless**. Specifying custom configurations for data tests will become available in dbt Core later this year. + +- **Enhancement**: Introducing a revamped overview page for dbt Explorer, available in beta. It includes a new design and layout for the Explorer homepage. The new layout provides a more intuitive experience for users to navigate their dbt projects, as well as a new **Latest updates** section to view the latest changes or issues related to project resources. To learn more, refer to [Overview page](/docs/collaborate/explore-projects#overview-page). + +#### dbt Semantic Layer +- **New**: Introduced the [`dbt-sl-sdk` Python software development kit (SDK)](https://github.com/dbt-labs/semantic-layer-sdk-python) Python library, which provides you with easy access to the dbt Semantic Layer with Python. It allows developers to interact with the dbt Semantic Layer APIs and query metrics and dimensions in downstream tools. Refer to the [dbt Semantic Layer Python SDK](/docs/dbt-cloud-apis/sl-python) for more information. - **New**: Introduced Semantic validations in CI pipelines. Automatically test your semantic nodes (metrics, semantic models, and saved queries) during code reviews by adding warehouse validation checks in your CI job using the `dbt sl validate` command. You can also validate modified semantic nodes to guarantee code changes made to dbt models don't break these metrics. Refer to [Semantic validations in CI](/docs/deploy/ci-jobs#semantic-validations-in-ci) to learn about the additional commands and use cases. +- **New**: We now expose the `meta` field within the [config property](/reference/resource-configs/meta) for dbt Semantic Layer metrics in the [JDBC and GraphQL APIs](/docs/dbt-cloud-apis/sl-api-overview) under the `meta` field. +- **New**: Added a new command in the dbt Cloud CLI called `export-all`, which allows you to export multiple or all of your saved queries. Previously, you had to explicitly specify the [list of saved queries](/docs/build/metricflow-commands#list-saved-queries). +- **Enhancement**: The dbt Semantic Layer now offers more granular control by supporting multiple data platform credentials, which can represent different roles or service accounts. Available for dbt Cloud Enterprise plans, you can map credentials to service tokens for secure authentication. Refer to [Set up dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) for more details. +- **Fix**: Addressed a bug where unicode query filters (such as Chinese characters) were not working correctly in the dbt Semantic Layer Tableau integration. +- **Fix**: Resolved a bug with parsing certain private keys for BigQuery when running an export. +- **Fix**: Addressed a bug that caused a "closed connection" error to be returned when querying or running an Export. +- **Fix**: Resolved an issue in dbt Core where, during partial parsing, all generated metrics in a file were incorrectly deleted instead of just those related to the changed semantic model. Now, only the metrics associated with the modified model are affected. ## June 2024 - **New:** Introduced new granularity support for cumulative metrics in MetricFlow. Granularity options for cumulative metrics are slightly different than granularity for other metric types. For other metrics, we use the `date_trunc` function to implement granularity. However, because cumulative metrics are non-additive (values can't be added up), we can't use the `date_trunc` function to change their time grain granularity. @@ -116,15 +135,15 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show - **New**: The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) introduces [declarative caching](/docs/use-dbt-semantic-layer/sl-cache), allowing you to cache common queries to speed up performance and reduce query compute costs. Available for dbt Cloud Team or Enterprise accounts. -- +- - The **Keep on latest version** setting is now Generally Available (previously Public Preview). + The **Versionless** setting is now Generally Available (previously Public Preview). - When the new **Keep on latest version** setting is enabled, you get a versionless experience and always get the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. + When the new **Versionless** setting is enabled, you get a versionless experience and always get the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. - To learn more about the new setting, refer to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) for details. + To learn more about the new setting, refer to [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) for details. - + @@ -140,7 +159,7 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show -- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and dbt Cloud "Keep on latest version"). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information. +- **Behavior change:** Introduced the `require_explicit_package_overrides_for_builtin_materializations` flag, opt-in and disabled by default. If set to `True`, dbt will only use built-in materializations defined in the root project or within dbt, rather than implementations in packages. This will become the default in May 2024 (dbt Core v1.8 and "Versionless" dbt Cloud). Read [Package override for built-in materialization](/reference/global-configs/legacy-behaviors#package-override-for-built-in-materialization) for more information. **dbt Semantic Layer** - **New**: Use Saved selections to [save your query selections](/docs/cloud-integrations/semantic-layer/gsheets#using-saved-selections) within the [Google Sheets application](/docs/cloud-integrations/semantic-layer/gsheets). They can be made private or public and refresh upon loading. @@ -195,15 +214,15 @@ The following features are new or enhanced as part of our [dbt Cloud Launch Show -- +- _Now available in the dbt version dropdown in dbt Cloud — starting with select customers, rolling out to wider availability through February and March._ - When the new **Keep on latest version** setting is enabled, you always get the latest fixes and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. + When the new **Versionless** setting is enabled, you always get the latest fixes and early access to new functionality for your dbt project. dbt Labs will handle upgrades behind-the-scenes, as part of testing and redeploying the dbt Cloud application — just like other dbt Cloud capabilities and other SaaS tools that you're using. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. - To learn more about the new setting, refer to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) for details. + To learn more about the new setting, refer to [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) for details. - + diff --git a/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md b/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md index 2de302993d3..0c725f129f8 100644 --- a/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md +++ b/website/docs/docs/dbt-versions/upgrade-dbt-version-in-cloud.md @@ -7,15 +7,15 @@ In dbt Cloud, both [jobs](/docs/deploy/jobs) and [environments](/docs/dbt-cloud- ## Environments -Navigate to the settings page of an environment, then click **Edit**. Click the **dbt version** dropdown bar and make your selection. You can select a previous release of dbt Core or go versionless by selecting [**Keep on latest version**](#keep-on-latest-version)(recommended). Be sure to save your changes before navigating away. +Navigate to the settings page of an environment, then click **Edit**. Click the **dbt version** dropdown bar and make your selection. You can select a previous release of dbt Core or go [**Versionless**](#versionless)(recommended). Be sure to save your changes before navigating away. -### Keep on latest version +### Versionless -By choosing to **Keep on latest version**, you opt for a versionless experience that provides the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades for you, as part of testing and redeploying the dbt Cloud SaaS application. Keep on latest version always includes the most recent version of dbt Core, and more. +By choosing to go **Versionless**, you opt for an experience that provides the latest features and early access to new functionality for your dbt project. dbt Labs will handle upgrades for you, as part of testing and redeploying the dbt Cloud SaaS application. Versionless always includes the most recent features before they're in dbt Core, and more. -You can upgrade to **Keep on latest version** and the versionless experience no matter which version of dbt you currently have selected. As a best practice, dbt Labs recommends that you test the upgrade in development first; use the [Override dbt version](#override-dbt-version) setting to test _your_ project on the latest dbt version before upgrading your deployment environments and the default development environment for all your colleagues. +You can upgrade to the **Versionless** experience no matter which version of dbt you currently have selected. As a best practice, dbt Labs recommends that you test the upgrade in development first; use the [Override dbt version](#override-dbt-version) setting to test _your_ project on the latest dbt version before upgrading your deployment environments and the default development environment for all your colleagues. To upgrade an environment in the [dbt Cloud Admin API](/docs/dbt-cloud-apis/admin-cloud-api) or [Terraform](https://registry.terraform.io/providers/dbt-labs/dbtcloud/latest), set `dbt_version` to the string `versionless`. diff --git a/website/docs/docs/dbt-versions/versionless-cloud.md b/website/docs/docs/dbt-versions/versionless-cloud.md index ae92e0d6a0a..17a97b0f319 100644 --- a/website/docs/docs/dbt-versions/versionless-cloud.md +++ b/website/docs/docs/dbt-versions/versionless-cloud.md @@ -1,18 +1,18 @@ --- -title: "Upgrade to \"Keep on latest version\" in dbt Cloud" -sidebar_label: "Upgrade to \"Keep on latest version\" " +title: "Upgrade to \"Versionless\" in dbt Cloud" +sidebar_label: "Upgrade to \"Versionless\" " description: "Learn how to go versionless in dbt Cloud. You never have to perform an upgrade again. Plus, you'll be able to access new features and enhancements as soon as they become available. " --- -dbt Cloud is going versionless. Soon, your environments and jobs will always run on the latest version of dbt. +dbt Cloud is going versionless. Soon, your environments and jobs will always run the latest features and functionality. This will require you to make one final update to your current jobs and environments. When that's done, you'll never have to think about managing, coordinating, or upgrading dbt versions again. -Move your environments and jobs to "Keep on latest version" to get all the functionality in the latest versions of dbt Core — and more! — along with access to the new features and fixes as soon as they’re released. +Move your environments and jobs to "Versionless" to get all the functionality in the latest features before they're in dbt Core — and more! — along with access to the new features and fixes as soon as they’re released. ## Tips for upgrading {#upgrade-tips} -If you regularly develop your dbt project in dbt Cloud and this is your first time trying “Keep on latest version,” dbt Labs recommends that you start in development because it will be the fastest for investigation and iteration. [Override your dbt version in development](/docs/dbt-versions/upgrade-dbt-version-in-cloud#override-dbt-version). Then, launch the IDE or Cloud CLI and do your development work as usual. Everything should work as you expect. +If you regularly develop your dbt project in dbt Cloud and this is your first time trying “Versionless,” dbt Labs recommends that you start in development because it will be the fastest for investigation and iteration. [Override your dbt version in development](/docs/dbt-versions/upgrade-dbt-version-in-cloud#override-dbt-version). Then, launch the IDE or Cloud CLI and do your development work as usual. Everything should work as you expect. If you do see something unexpected or surprising, revert back to the previous version and record the differences you observed. [Contact dbt Cloud support](/docs/dbt-support#dbt-cloud-support) with your findings for a more detailed investigation. @@ -24,7 +24,7 @@ If your organization has multiple dbt projects, we recommend starting your upgra The following is our guidance on some important considerations regarding dbt projects as part of the upgrade. -To learn more about how dbt Labs deploys stable dbt upgrades in a safe manner to dbt Cloud, we recommend that you read our blog post [How we're making sure you can confidently "Keep on latest version" in dbt Cloud](https://docs.getdbt.com/blog/latest-dbt-stability) for details. +To learn more about how dbt Labs deploys stable dbt upgrades in a safe manner to dbt Cloud, we recommend that you read our blog post [How we're making sure you can confidently go "Versionless" in dbt Cloud](https://docs.getdbt.com/blog/latest-dbt-stability) for details. @@ -44,9 +44,9 @@ When we talk about _latest version_, we’re referring to the underlying runtime If a new version of a dbt package includes a breaking change (for example, a change to one of the macros in `dbt_utils`), you don’t have to immediately use the new version. In your `packages` configuration (in `dependencies.yml` or `packages.yml`), you can still specify which versions or version ranges of packages you want dbt to install. If you're not already doing so, we strongly recommend [checking `package-lock.yml` into version control](/reference/commands/deps#predictable-package-installs) for predictable package installs in deployment environments and a clear change history whenever you install upgrades. -If you upgrade to “Keep on latest version” and immediately see something that breaks, please [contact support](/docs/dbt-support#dbt-cloud-support) and, in the meantime, downgrade back to v1.7. +If you upgrade to “Versionless” and immediately see something that breaks, please [contact support](/docs/dbt-support#dbt-cloud-support) and, in the meantime, downgrade back to v1.7. -If you’re already on “Keep on latest version” and you observe a breaking change (like something worked yesterday, but today it isn't working, or works in a surprising/different way), please [contact support](/docs/dbt-support#dbt-cloud-support) immediately. Depending on your contracted support agreement, the dbt Labs team will respond within our SLA time and we would seek to roll back the change and/or roll out a fix (just as we would for any other part of dbt Cloud). This is the same whether or not the root cause of the breaking change is in the project code or in the code of a package. +If you’re already on “Versionless” and you observe a breaking change (like something worked yesterday, but today it isn't working, or works in a surprising/different way), please [contact support](/docs/dbt-support#dbt-cloud-support) immediately. Depending on your contracted support agreement, the dbt Labs team will respond within our SLA time and we would seek to roll back the change and/or roll out a fix (just as we would for any other part of dbt Cloud). This is the same whether or not the root cause of the breaking change is in the project code or in the code of a package. If the package you’ve installed relies on _undocumented_ functionality of dbt, it doesn't have the same guarantees as functionality that we’ve documented and tested. However, we will still do our best to avoid breaking them. @@ -54,11 +54,11 @@ If the package you’ve installed relies on _undocumented_ functionality of db -No. Going forward, “Keep on latest version” will be how all customers are going to access new functionality and ongoing support in dbt Cloud. We believe this is the best way for us to offer a reliable, stable, and secure runtime for dbt with continuous and consistent updates. +No. Going forward, “Versionless” will be how all customers are going to access new functionality and ongoing support in dbt Cloud. We believe this is the best way for us to offer a reliable, stable, and secure runtime for dbt with continuous and consistent updates. In 2023 (and earlier), customers were expected to manage their own upgrades by selecting dbt Core versions, up to and including dbt Core v1.7, which was released in October 2023. (Way back in 2021, dbt Cloud customers would pick specific _patch releases_ of dbt Core, such as upgrading from `v0.21.0` to `v0.21.1`. We’ve come a long way since then!) -In 2024, we've changed the way that new dbt functionality is made available for dbt Cloud customers: continuously. Behavior or breaking changes are gated behind opt-in flags. Users don't need to spend valuable time managing their own upgrades. This is called "Keep on latest version" and it's required for accessing any new functionality that we've put out in 2024+. +In 2024, we've changed the way that new dbt functionality is made available for dbt Cloud customers: continuously. Behavior or breaking changes are gated behind opt-in flags. Users don't need to spend valuable time managing their own upgrades. This is called "Versionless" dbt Cloud and it's required for accessing any new functionality that we've put out in 2024+. We will absolutely continue to release new minor versions of dbt Core (OSS), including v1.9 which will be available later this year. When we do, it will be a subset of the functionality that's already available to dbt Cloud customers and always after the functionality has been available in dbt Cloud. diff --git a/website/docs/docs/deploy/continuous-deployment.md b/website/docs/docs/deploy/continuous-deployment.md new file mode 100644 index 00000000000..02144e0040e --- /dev/null +++ b/website/docs/docs/deploy/continuous-deployment.md @@ -0,0 +1,20 @@ +--- +title: "Continuous deployment in dbt Cloud" +sidebar_label: "Continuous deployment" +description: "Learn about continuous deployment (CD) workflows " +--- + +To help you improve data transformations and ship data products faster, you can run [merge jobs](/docs/deploy/merge-jobs) to implement a continuous deployment (CD) workflow in dbt Cloud. Merge jobs can automatically build modified models whenever a pull request (PR) merges, making sure the latest code changes are in production. You don't have to wait for the next scheduled job to run to get the latest updates. + +You can also implement continuous integration (CI) in dbt Cloud, which can help further to reduce the time it takes to push changes to production and improve code quality. To learn more, refer to [Continuous integration in dbt Cloud](/docs/deploy/continuous-integration). + +## How merge jobs work + +When you set up merge jobs, dbt Cloud listens for notifications from your [Git provider](/docs/cloud/git/git-configuration-in-dbt-cloud) indicating that a PR has been merged. When dbt Cloud receives one of these notifications, it enqueues a new run of the merge job. + +You can set up merge jobs to perform one of the following when a PR merges: + +|
Command to run
| Usage description | +| -------- | ----------------- | +| `dbt build --select state:modified+` | (Default) Build the modified data with every merge.

dbt Cloud builds only the changed data models and anything downstream of it, similar to CI jobs. This helps reduce computing costs and ensures that the latest code changes are always pushed to production. | +| `dbt compile` | Refresh the applied state for performant (the slimmest) CI job runs.

dbt Cloud generates the executable SQL (from the source model, test, and analysis files) but does not run it. This ensures the changes are reflected in the manifest for the next time a CI job is run and keeps track of only the relevant changes. | diff --git a/website/docs/docs/deploy/continuous-integration.md b/website/docs/docs/deploy/continuous-integration.md index bf27f68a863..fbe93e084b6 100644 --- a/website/docs/docs/deploy/continuous-integration.md +++ b/website/docs/docs/deploy/continuous-integration.md @@ -50,6 +50,6 @@ When you push a new commit to a PR, dbt Cloud enqueues a new CI run for the late -### Run slot treatment +### Run slot treatment -For accounts on the [Enterprise or Team](https://www.getdbt.com/pricing) plans, CI runs won't consume run slots. This guarantees a CI check will never block a production run. +CI runs don't consume run slots. This guarantees a CI check will never block a production run. diff --git a/website/docs/docs/deploy/deploy-environments.md b/website/docs/docs/deploy/deploy-environments.md index 50d1b7ac99e..088ecb0d841 100644 --- a/website/docs/docs/deploy/deploy-environments.md +++ b/website/docs/docs/deploy/deploy-environments.md @@ -57,13 +57,10 @@ Some customers prefer to connect Development and Staging to their `main` branch ### Why use a staging environment -There are two primary motivations for using a Staging environment: +These are the primary motivations for using a Staging environment: 1. An additional validation layer before changes are deployed into Production. You can deploy, test, and explore your dbt models in Staging. 2. Clear isolation between development workflows and production data. It enables developers to work in metadata-powered ways, using features like deferral and cross-project references, without accessing data in production deployments. - -:::info Coming soon: environment-level permissions -Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down. -::: +3. Provide developers with the ability to create, edit, and trigger ad hoc jobs in the Staging environment, while keeping the Production environment locked down using [environment-level permissions](/docs/cloud/manage-access/environment-permissions). **Conditional configuration of sources** enables you to point to "prod" or "non-prod" source data, depending on the environment you're running in. For example, this source will point to `.sensitive_source.table_with_pii`, where `` is dynamically resolved based on an environment variable. diff --git a/website/docs/docs/deploy/deployment-overview.md b/website/docs/docs/deploy/deployment-overview.md index 2e90fdc8d05..539b083e42e 100644 --- a/website/docs/docs/deploy/deployment-overview.md +++ b/website/docs/docs/deploy/deployment-overview.md @@ -44,9 +44,9 @@ Learn how to use dbt Cloud's features to help your team ship timely and quality icon="dbt-bit"/> - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer, powered by [MetricFlow](/docs/build/about-metricflow), simplifies the process of defining and using critical business metrics, like `revenue` in the modeling layer (your dbt project). By centralizing metric definitions, data teams can ensure consistent self-service access to these metrics in downstream data tools and applications. The dbt Semantic Layer eliminates duplicate coding by allowing data teams to define metrics on top of existing models and automatically handles data joins. -Moving metric definitions out of the BI layer and into the modeling layer allows data teams to feel confident that different business units are working from the same metric definitions, regardless of their tool of choice. If a metric definition changes in dbt, it’s refreshed everywhere it’s invoked and creates consistency across all applications. +Moving metric definitions out of the BI layer and into the modeling layer allows data teams to feel confident that different business units are working from the same metric definitions, regardless of their tool of choice. If a metric definition changes in dbt, it’s refreshed everywhere it’s invoked and creates consistency across all applications. To ensure secure access control, the dbt Semantic Layer implements robust [access permissions](/docs/use-dbt-semantic-layer/setup-sl#set-up-dbt-semantic-layer) mechanisms. Refer to the [dbt Semantic Layer FAQs](/docs/use-dbt-semantic-layer/sl-faqs) or [Why we need a universal semantic layer](https://www.getdbt.com/blog/universal-semantic-layer/) blog post to learn more. diff --git a/website/docs/docs/use-dbt-semantic-layer/exports.md b/website/docs/docs/use-dbt-semantic-layer/exports.md index a563df40ef7..02480ad6617 100644 --- a/website/docs/docs/use-dbt-semantic-layer/exports.md +++ b/website/docs/docs/use-dbt-semantic-layer/exports.md @@ -21,6 +21,32 @@ Essentially, exports are like any other table in your data platform — they - You have a [saved query](/docs/build/saved-queries) and [export configured](/docs/build/saved-queries#configure-exports) in your dbt project. In your configuration, leverage [caching](/docs/use-dbt-semantic-layer/sl-cache) to cache common queries, speed up performance, and reduce compute costs. - You have the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) installed. Note, that exports aren't supported in dbt Cloud IDE yet. +## Benefits of exports + +The following section explains the main benefits of using exports, including: +- [DRY representation](#dry-representation) +- [Easier changes](#easier-changes) +- [Caching](#caching) + +#### DRY representation + +Currently, creating tables often involves generating tens, hundreds, or even thousands of tables that denormalize data into summary or metric mart tables. The main benefit of exports is creating a "Don't Repeat Yourself (DRY)" representation of the logic to construct each metric, dimension, join, filter, and so on. This allows you to reuse those components for long-term scalability, even if you're replacing manually written SQL models with references to the metrics or dimensions in saved queries. + +#### Easier changes + +Exports ensure that changes to metrics and dimensions are made in one place and then cascade to those various destinations seamlessly. This prevents the problem of needing to update a metric across every model that references that same concept. + +#### Caching +Use exports to pre-populate the cache, so that you're pre-computing what you need to serve users through the dynamic Semantic Layer APIs. + +#### Considerations + +Exports offer many benefits and it's important to note some use cases that fall outside the advantages: +- Business users may still struggle to consume from tens, hundreds, or thousands of tables, and choosing the right one can be a challenge. +- Business users may also make mistakes when aggregating and filtering from the pre-built tables. + +For these use cases, use the dynamic [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) instead of exports. + ## Run exports Before you're able to run exports in development or production, you'll need to make sure you've [configured saved queries and exports](/docs/build/saved-queries) in your dbt project. In your saved query config, you can also leverage [caching](/docs/use-dbt-semantic-layer/sl-cache) with the dbt Cloud job scheduler to cache common queries, speed up performance, and reduce compute costs. @@ -32,7 +58,17 @@ There are two ways to run an export: ## Exports in development -You can run an export in your development environment using your development credentials if you want to test the output of the export before production. You can use the following command to run exports in the dbt Cloud CLI: +You can run an export in your development environment using your development credentials if you want to test the output of the export before production. + +This section explains the different commands and options available to run exports in development. + +- Use the [`dbt sl export` command](#exports-for-single-saved-query) to test and generate exports in your development environment for a singular saved query. You can also use the `--select` flag to specify particular exports from a saved query. + +- Use the [`dbt sl export-all` command](#exports-for-multiple-saved-queries) to run exports for multiple saved queries at once. This command provides a convenient way to manage and execute exports for several queries simultaneously, saving time and effort. + +### Exports for single saved query + +Use the following command to run exports in the dbt Cloud CLI: ```bash dbt sl export @@ -52,13 +88,11 @@ The following table lists the options for `dbt sl export` command, using the `-- You can also run any export defined for the saved query and write the table or view in your development environment. Refer to the following command example and output: -#### Example - ```bash dbt sl export --saved-query sq_name ``` -#### Output +The output would look something like this: ```bash Polling for export status - query_id: 2c1W6M6qGklo1LR4QqzsH7ASGFs.. @@ -93,6 +127,29 @@ dbt sl export --saved-query sq_number1 --export-as table --alias new_export ``` +### Exports for multiple saved queries + +Use the command, `dbt sl export-all`, to run exports for multiple saved queries at once. This is different from the `dbt sl export` command, which only runs exports for a singular saved query. For example, to run exports for multiple saved queries, you can use: + +```bash +dbt sl export-all +``` + +The output would look something like this: + +```bash +Exports completed: +- Created TABLE at `DBT_SL_TEST.new_customer_orders` +- Created VIEW at `DBT_SL_TEST.new_customer_orders_export_alias` +- Created TABLE at `DBT_SL_TEST.order_data_key_metrics` +- Created TABLE at `DBT_SL_TEST.weekly_revenue` + +Polling completed +``` + +The command `dbt sl export-all` provides the flexibility to manage multiple exports in a single command. + + ## Exports in production Enabling and executing exports in dbt Cloud optimizes data workflows and ensures real-time data access. It enhances efficiency and governance for smarter decisions. @@ -118,7 +175,7 @@ If exports aren't needed, you can set the value(s) to `FALSE` (`DBT_INCLUDE_SAVE - + 1. Click **Deploy** in the top navigation bar and choose **Environments**. diff --git a/website/docs/docs/use-dbt-semantic-layer/setup-sl.md b/website/docs/docs/use-dbt-semantic-layer/setup-sl.md index ae185a0343a..03dc605d83c 100644 --- a/website/docs/docs/use-dbt-semantic-layer/setup-sl.md +++ b/website/docs/docs/use-dbt-semantic-layer/setup-sl.md @@ -6,15 +6,7 @@ sidebar_label: "Set up your Semantic Layer" tags: [Semantic Layer] --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - -With the dbt Semantic Layer, you can centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. Configure the dbt Semantic Layer in dbt Cloud to connect with your integrated partner tool. +With the dbt Semantic Layer, you can centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. ## Prerequisites @@ -49,10 +41,14 @@ import SlSetUp from '/snippets/_new-sl-setup.md'; - [Optimize querying performance](/docs/use-dbt-semantic-layer/sl-cache) using declarative caching. - [Validate semantic nodes in CI](/docs/deploy/ci-jobs#semantic-validations-in-ci) to ensure code changes made to dbt models don't break these metrics. - If you haven't already, learn how to [build you metrics and semantic models](/docs/build/build-metrics-intro) in your development tool of choice. +- Learn about commonly asked [dbt Semantic Layer FAQs](/docs/use-dbt-semantic-layer/sl-faqs). + +## FAQs + + + +Cached data is stored separately from the underlying models. If metrics are pulled from the cache, we don’t have the security context applied to those tables at query time. -## Related docs +In the future, we plan to clone credentials, identify the minimum access level needed, and apply those permissions to cached tables. -- [Build your metrics](/docs/build/build-metrics-intro) -- [Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) -- [Get started with the dbt Semantic Layer](/guides/sl-snowflake-qs) -- [dbt Semantic Layer FAQs](/docs/use-dbt-semantic-layer/sl-faqs) + diff --git a/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md b/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md index 5f0c50b3e43..d3b79337111 100644 --- a/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md +++ b/website/docs/docs/use-dbt-semantic-layer/sl-architecture.md @@ -7,14 +7,6 @@ tags: [Semantic Layer] pagination_next: null --- - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - The dbt Semantic Layer allows you to define metrics and use various interfaces to query them. The Semantic Layer does the heavy lifting to find where the queried data exists in your data platform and generates the SQL to make the request (including performing joins). diff --git a/website/docs/docs/use-dbt-semantic-layer/sl-cache.md b/website/docs/docs/use-dbt-semantic-layer/sl-cache.md index 5f1460b07f5..99ae014939a 100644 --- a/website/docs/docs/use-dbt-semantic-layer/sl-cache.md +++ b/website/docs/docs/use-dbt-semantic-layer/sl-cache.md @@ -22,7 +22,7 @@ While you can use caching to speed up your queries and reduce compute time, know ## Prerequisites - dbt Cloud [Team or Enterprise](https://www.getdbt.com/) plan. -- dbt Cloud environments that are versionless by opting to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). +- dbt Cloud environments that are ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless). - A successful job run and [production environment](/docs/deploy/deploy-environments#set-as-production-environment). - For declarative caching, you need to have [exports](/docs/use-dbt-semantic-layer/exports) defined in your [saved queries](/docs/build/saved-queries) YAML configuration file. @@ -132,6 +132,16 @@ If an upstream model has data in it that was created after the cache was created You can manually invalidate the cache through the [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview) using the `InvalidateCacheResult` field. +## FAQs + + + +Cached data is stored separately from the underlying models. If metrics are pulled from the cache, we don’t have the security context applied to those tables at query time. + +In the future, we plan to clone credentials, identify the minimum access level needed, and apply those permissions to cached tables. + + + ## Related docs - [Validate semantic nodes in CI](/docs/deploy/ci-jobs#semantic-validations-in-ci) diff --git a/website/docs/docs/use-dbt-semantic-layer/sl-faqs.md b/website/docs/docs/use-dbt-semantic-layer/sl-faqs.md index fb7ed58ba0d..9a122aa644a 100644 --- a/website/docs/docs/use-dbt-semantic-layer/sl-faqs.md +++ b/website/docs/docs/use-dbt-semantic-layer/sl-faqs.md @@ -269,9 +269,12 @@ Yes, all of our interfaces or APIs expose metric descriptions, which you can sur +The dbt Semantic Layer uses service tokens for authentication, mapped to underlying data platform credentials. These credentials control physical access to the raw data. The credential configuration allows admins to create a credential and map it to service tokens, which can then be shared to relevant teams for BI connection setup. You can configure credentials and service tokens to reflect your teams and their roles. + Currently, the credentials you configure when setting up the dbt Semantic Layer are used for every request. Any physical access policies you have tied to your credentials will be respected. We are currently working on introducing more fine-grained access controls, including user-level access and group credentials, that enable flexible granular permissions. + ## Implementation diff --git a/website/docs/faqs/API/rotate-token.md b/website/docs/faqs/API/rotate-token.md index d21db6f6165..8dea2d0b875 100644 --- a/website/docs/faqs/API/rotate-token.md +++ b/website/docs/faqs/API/rotate-token.md @@ -34,23 +34,21 @@ curl --location --request POST 'https://cloud.getdbt.com/api/v3/accounts/YOUR_AC * Find your `YOUR_CURRENT_TOKEN` by going to **Profile Settings** -> **API Access** and copying the API key. * Find [`YOUR_ACCESS_URL`](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan. - -For example, if `YOUR_ACCOUNT_ID` = `000`, `YOUR_USER_ID` = `123`, `YOUR_CURRENT_PAT_TOKEN` = `abcf9g`, and your `ACCESS_URL` = `cloud.getdbt.com`, then your curl request will be: +If `YOUR_USER_ID` = `123`, `YOUR_CURRENT_TOKEN` = `abcf9g`, then your curl request will be: ``` -curl --location --request POST 'https://cloud.getdbt.com/api/v3/accounts/000/users/123/apikey/' \ +curl --location --request POST 'https://YOUR_ACCESS_URL/api/v2/users/123/apikey/' \ + --header 'Authorization: Token abcf9g' ``` - 2. Find the new key in the API response or in dbt Cloud. 3. To find the new key in dbt Cloud, go to **Profile Settings** -> **API Access**. - ### dbt Cloud deployments -If your [dbt Cloud deployment](/docs/cloud/about-cloud/access-regions-ip-addresses) uses a different access URL, replace `cloud.getdbt.com` with the URL of your instance. +If your [dbt Cloud deployment](/docs/cloud/about-cloud/access-regions-ip-addresses) uses a different access URL, replace `YOUR_ACCESS_URL` with the URL of your instance. For example, if your deployment is Virtual Private dbt: diff --git a/website/docs/faqs/Accounts/change-billing.md b/website/docs/faqs/Accounts/change-billing.md index 65e47cc9c4a..11290728c98 100644 --- a/website/docs/faqs/Accounts/change-billing.md +++ b/website/docs/faqs/Accounts/change-billing.md @@ -6,6 +6,6 @@ id: change-billing --- -If you want to change your account's credit card details, select the gear menu in the upper right corner of dbt Cloud. Go to Account Settings → Billing → Payment Information. In the upper right corner of Payment Information, click **Edit** to enter the new credit card details. Only the _account owner_ can make this change. +If you want to change your account's credit card details, select the gear menu in the upper right corner of dbt Cloud. Go to Account Settings → Billing → Payment Information. Enter the new credit card details on the respective fields then click on **Update payment information**. Only the _account owner_ can make this change. To change your billing name or location address, send our Support team a message at support@getdbt.com with the newly updated information, and we can make that change for you! diff --git a/website/docs/faqs/Accounts/find-user-id.md b/website/docs/faqs/Accounts/find-user-id.md index 7f7eca2cbba..09e3ed35a0b 100644 --- a/website/docs/faqs/Accounts/find-user-id.md +++ b/website/docs/faqs/Accounts/find-user-id.md @@ -14,4 +14,4 @@ To find your user ID in dbt Cloud, read the following steps: 3. In the address bar, the number after `/users` is your user ID. 4. Copy that number or save it somewhere safe.
-For example, if the URL is `https://cloud.getdbt.com/settings/accounts/12345/users/67891` — the user ID is `67891`

\ No newline at end of file +For example, if the URL is `https://YOUR_ACCESS_URL/settings/accounts/12345/users/67891` — the user ID is `67891`

\ No newline at end of file diff --git a/website/docs/faqs/Core/install-pip-os-prereqs.md b/website/docs/faqs/Core/install-pip-os-prereqs.md index 1eb6205512a..c8435b44f33 100644 --- a/website/docs/faqs/Core/install-pip-os-prereqs.md +++ b/website/docs/faqs/Core/install-pip-os-prereqs.md @@ -23,13 +23,7 @@ sudo yum install redhat-rpm-config gcc libffi-devel \ ### MacOS - The MacOS requires Python 3.8 or higher to successfully install and run dbt Core. - - - -The MacOS requires Python 3.7 or higher to successfully install and run dbt Core. - To check the Python version: @@ -61,15 +55,6 @@ pip install cryptography~=3.4 Windows requires Python and git to successfully install and run dbt Core. - - Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.8 or higher for Windows](https://www.python.org/downloads/windows/). - - - - - -Install [Git for Windows](https://git-scm.com/downloads) and [Python version 3.7 or higher for Windows](https://www.python.org/downloads/windows/). - For further questions, please see the [Python compatibility FAQ](/faqs/Core/install-python-compatibility) diff --git a/website/docs/faqs/Git/github-permissions.md b/website/docs/faqs/Git/github-permissions.md index e3a1740bbab..075343e0c5e 100644 --- a/website/docs/faqs/Git/github-permissions.md +++ b/website/docs/faqs/Git/github-permissions.md @@ -6,29 +6,41 @@ sidebar_label: "GitHub and dbt Cloud permissions error" If you see the error `This account needs to accept the latest permissions for the dbt Cloud GitHub App` in dbt Cloud — this usually occurs when the permissions for the dbt Cloud GitHub App are out of date. -To solve this issue, you'll need to update the permissions for the dbt Cloud GitHub App in your GitHub account. Here's a couple of ways you can do it: +To solve this issue, you'll need to update the permissions for the dbt Cloud GitHub App in your GitHub account. This FAQ shares a couple of ways you can do it. -#### Update permissions +## Update permissions -A Github organization admin will need to update the permissions in GitHub for the dbt Cloud GitHub App. If you're not the admin, reach out to your organization admin to request this. Alternatively, try [disconecting your GitHub account](#disconect-github) in dbt Cloud. +A GitHub organization admin will need to update the permissions in GitHub for the dbt Cloud GitHub App. If you're not the admin, reach out to your organization admin to request this. + +1. Navigate to your GitHub account. Click on the top right profile icon and then **Settings** (or personal if using a non-organization account). + + + +2. Then go to **Integrations** and then select **Applications** to identify any necessary permission changes. Note that a GitHub repository admin may not see the same permission request. + + + +3. Click on **Review request** and then click on the **Accept new permissions** button on the next page. + + -1. Go directly to GitHub to determine if any updated permissions are required. -2. In GitHub, go to your organization **Settings** (or personal if using a non-organization account). -3. Then navigate to **Applications** to identify any necessary permission changes. For more info on GitHub permissions, refer to [access permissions](https://docs.github.com/en/get-started/learning-about-github/access-permissions-on-github). -#### Disconnect GitHub +Alternatively, try [disconecting your GitHub account](#disconect-github) in dbt Cloud, detailed in the following section. + +## Disconnect GitHub Disconnect the GitHub and dbt Cloud integration in dbt Cloud. 1. In dbt Cloud, go to **Account Settings**. -2. In **Projects**, select the project that's experiencing the issue. +2. In **Projects**, select the project experiencing the issue. 3. Click the repository link under **Repository**. 4. In the **Repository details** page, click **Edit**. 5. Click **Disconnect** to remove the GitHub integration. -6. Go back to your **Project details** page and reconnect your repository by clicking the **Configure Repository** link. +6. Return to your **Project details** page and reconnect your repository by clicking the **Configure Repository** link. 7. Configure your repository and click **Save** - + +## Support If you've tried these workarounds and are still experiencing this behavior — reach out to the [dbt Support](mailto:support@getdbt.com) team and we'll be happy to help! diff --git a/website/docs/faqs/Git/gitignore.md b/website/docs/faqs/Git/gitignore.md index 8d966c40e2c..16575861289 100644 --- a/website/docs/faqs/Git/gitignore.md +++ b/website/docs/faqs/Git/gitignore.md @@ -13,7 +13,6 @@ If you encounter issues like problems reverting changes, checking out or creatin To resolve issues with your `gitignore` file, adding the correct entries won't automatically remove (or 'untrack') files or folders that have already been tracked by git. The updated `gitignore` will only prevent new files or folders from being tracked. So you'll need to first fix the `gitignore` file, then perform some additional git operations to untrack any incorrect files or folders. - 1. Launch the Cloud IDE into the project that is being fixed, by selecting **Develop** on the menu bar. 2. In your **File Explorer**, check to see if a `.gitignore` file exists at the root of your dbt project folder. If it doesn't exist, create a new file. @@ -121,6 +120,4 @@ dbt_modules/ - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/faqs/Models/unique-model-names.md b/website/docs/faqs/Models/unique-model-names.md index 7878a5a704c..6d8bd18ac00 100644 --- a/website/docs/faqs/Models/unique-model-names.md +++ b/website/docs/faqs/Models/unique-model-names.md @@ -6,20 +6,8 @@ id: unique-model-names --- - - Within one project: yes! To build dependencies between models, you need to use the `ref` function, and pass in the model name as an argument. dbt uses that model name to uniquely resolve the `ref` to a specific model. As a result, these model names need to be unique, _even if they are in distinct folders_. A model in one project can have the same name as a model in another project (installed as a dependency). dbt uses the project name to uniquely identify each model. We call this "namespacing." If you `ref` a model with a duplicated name, it will resolve to the model within the same namespace (package or project), or raise an error because of an ambiguous reference. Use [two-argument `ref`](/reference/dbt-jinja-functions/ref#ref-project-specific-models) to disambiguate references by specifying the namespace. Those models will still need to land in distinct locations in the data warehouse. Read the docs on [custom aliases](/docs/build/custom-aliases) and [custom schemas](/docs/build/custom-schemas) for details on how to achieve this. - - - - - -Yes! To build dependencies between models, you need to use the `ref` function, and pass in the model name as an argument. dbt uses that model name to uniquely resolve the `ref` to a specific model. As a result, these model names need to be unique, _even if they are in distinct folders_. - -Often, this question comes up because users want to give two models the same name in their warehouse, splitting them across separate schemas (e.g. `stripe.users` and `app.users`). Checkout the docs on [custom aliases](/docs/build/custom-aliases) and [custom schemas](/docs/build/custom-schemas) to achieve this. - - diff --git a/website/docs/faqs/Project/why-version-2.md b/website/docs/faqs/Project/why-version-2.md index b4e91d6a773..6318ccf037d 100644 --- a/website/docs/faqs/Project/why-version-2.md +++ b/website/docs/faqs/Project/why-version-2.md @@ -6,11 +6,7 @@ id: why-version-2 --- - - Once upon a time, the structure of these `.yml` files was very different (s/o to anyone who was using dbt back then!). Adding `version: 2` allowed us to make this structure more extensible. Resource yml files do not currently require this config. We only support `version: 2` if it's specified. Although we do not expect to update yml files to `version: 3` soon, having this config will make it easier for us to introduce new structures in the future - - diff --git a/website/docs/faqs/Snapshots/snapshot-target-schema.md b/website/docs/faqs/Snapshots/snapshot-target-schema.md deleted file mode 100644 index 70c5b56f067..00000000000 --- a/website/docs/faqs/Snapshots/snapshot-target-schema.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Why is there only one `target_schema` for snapshots? -description: "Snapshots build into the same `target_schema`" -sidebar_label: 'Build snapshots into one `target_schema`' -id: snapshot-target-schema - ---- - -Snapshots build into the same `target_schema`, no matter who is running them. - -In comparison, models build into a separate schema for each user — this helps maintain separate development and production environments. - -So, why the difference? - -Let's assume you are running your snapshot regularly. If the model had a different target in `dev` (e.g. `dbt_claire`) compared to `prod` (e.g. `analytics`), when you `ref` the model in `dev`, dbt would select from a snapshot that has not been run regularly. This can make it hard to build models since the data differs from prod. - -Instead, in the models that `ref` your snapshots, it makes more sense to `select` from the production version of your snapshot, even when developing models. In this way, snapshot tables are more similar to source data than they are to proper dbt models. - -For this reason, there is only _one_ `target_schema`, which is _not_ environment-aware by default. - -However, this can create problems if you need to run a `snapshot` command when developing your models, or during a CI run. Fortunately, there's a few workarounds — check out [this forum article](https://discourse.getdbt.com/t/using-dynamic-schemas-for-snapshots/1070). diff --git a/website/docs/faqs/Troubleshooting/gitignore.md b/website/docs/faqs/Troubleshooting/gitignore.md index 6ab217ebf07..1519ef99ee3 100644 --- a/website/docs/faqs/Troubleshooting/gitignore.md +++ b/website/docs/faqs/Troubleshooting/gitignore.md @@ -11,8 +11,6 @@ If you can't revert changes, check out a branch, or click commit — this is To fix this, complete the following steps: - - 1. In the dbt Cloud IDE, add the following [.gitignore contents](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) in your dbt project `.gitignore` file: ```bash target/ @@ -37,7 +35,4 @@ dbt_modules/ - - - For more info, refer to this [detailed video](https://www.loom.com/share/9b3b8e2b617f41a8bad76ec7e42dd014) for additional guidance. diff --git a/website/docs/faqs/Troubleshooting/job-memory-limits.md b/website/docs/faqs/Troubleshooting/job-memory-limits.md index cfbecf9e25d..06f6a752507 100644 --- a/website/docs/faqs/Troubleshooting/job-memory-limits.md +++ b/website/docs/faqs/Troubleshooting/job-memory-limits.md @@ -15,12 +15,16 @@ Some common reasons for higher memory usage are: ## Resolution -Try the following to resolve this: +There are various reasons why you could be experiencing this error but they are mostly the outcome of retrieving too much data back into dbt. For example, using the `run_query()` operations or similar macros, or even using database/schemas that have a lot of other non-dbt related tables/views. Try to reduce the amount of data / number of rows retrieved back into dbt by refactoring the SQL in your `run_query()` operation using `group`, `where`, or `limit` clauses. Additionally, you can also use a database/schema with fewer non-dbt related tables/views. -1. **Use incremental models**: Try using [incremental models](/docs/build/incremental-models-overview) to reduce the amount of data being processed in each run. Incremental models only process new or updated data, which can help reduce the memory usage of your jobs. -2. **Refactor your data model**: Review your data models to see if there are any opportunities to optimize or refactor them. For example, you can try to reduce the number of columns being selected, use `where` clauses to filter data early in the query or use `limit` clauses to reduce the amount of data being processed. -If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com) and we can try increasing your account's memory. We're happy to help! + +:::tip Video example +As an additional resource, check out [this example video](https://www.youtube.com/watch?v=sTqzNaFXiZ8), which demonstrates how to refactor the sample code by reducing the number of rows returned. +::: + + +If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com). We're happy to help! ## Additional resources - [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) diff --git a/website/docs/guides/adapter-creation.md b/website/docs/guides/adapter-creation.md index 20d7bae26e2..b6528495260 100644 --- a/website/docs/guides/adapter-creation.md +++ b/website/docs/guides/adapter-creation.md @@ -76,12 +76,12 @@ Differences between databases are encoded into discrete areas: | Components | Code Path | Function | |------------------|---------------------------------------------------|-------------------------------------------------------------------------------| -| Python Classes | `adapters/` | Configuration (See above [Python classes](##python classes) | +| Python classes | `adapters/` | Configuration (Refer to [Python classes](#python classes) | | Macros | `include//macros/adapters/` | SQL API & statement syntax (for example, how to create schema or how to get table info) | | Materializations | `include//macros/materializations/` | Table/view/snapshot/ workflow definitions | -#### Python Classes +#### Python classes These classes implement all the methods responsible for: - Connecting to a database and issuing queries. @@ -1290,14 +1290,13 @@ The adapter should have the required documentation for connecting and configurin Proceed to the "Document a new adapter" step for more information. -### Release Cadence +### Release cadence -Keeping an adapter up-to-date with the latest features of dbt Core, as defined in [dbt-adapters](https://github.com/dbt-labs/dbt-adapters), is an integral part of being a trusted adapter. Therefore, we ask that adapter maintainers: +Keeping an adapter up-to-date with the latest features of dbt, as defined in [dbt-adapters](https://github.com/dbt-labs/dbt-adapters), is an integral part of being a trusted adapter. We encourage adapter maintainers to keep track of new dbt-adapter releases and support new features relevant to their platform, ensuring users have the best version of dbt. -- Release of new minor versions of the adapter with all tests passing within four weeks of dbt Core's release cut. -- Release of new major versions of the adapter with all tests passing within eight weeks of dbt Core's release cut. +Before [dbt Core version 1.8](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#new-dbt-core-adapter-installation-procedure), adapter versions needed to match the semantic versioning of dbt Core. After v1.8, this is no longer required. This means users can use an adapter on v1.8+ with a different version of dbt Core v1.8+. For example, a user could use dbt-core v1.9 with dbt-postgres v1.8. -### Community Responsiveness +### Community responsiveness On a best effort basis, active participation and engagement with the dbt Community across the following forums: @@ -1346,4 +1345,4 @@ The approval workflow is as follows: Ask your question in #adapter-ecosystem channel of the dbt community Slack. - \ No newline at end of file + diff --git a/website/docs/guides/airflow-and-dbt-cloud.md b/website/docs/guides/airflow-and-dbt-cloud.md index 0f1ea0f425f..51ac7668aa9 100644 --- a/website/docs/guides/airflow-and-dbt-cloud.md +++ b/website/docs/guides/airflow-and-dbt-cloud.md @@ -117,7 +117,7 @@ cd airflow-dbt-cloud - Once you hit `save` on the job, make sure you copy the URL and save it for referencing later. The url will look similar to this: ```html -https://cloud.getdbt.com/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ +https://YOUR_ACCESS_URL/#/accounts/{account_id}/projects/{project_id}/jobs/{job_id}/ ``` @@ -134,7 +134,7 @@ Now you have all the working pieces to get up and running with Airflow + dbt Clo ![Connection type](/img/guides/orchestration/airflow-and-dbt-cloud/connection-type.png) -3. Add in your connection details and your default dbt Cloud account id. This is found in your dbt Cloud URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/` +3. Add in your connection details and your default dbt Cloud account id. This is found in your dbt Cloud URL after the accounts route section (`/accounts/{YOUR_ACCOUNT_ID}`), for example the account with id 16173 would see this in their URL: `https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/` ![Connection type](/img/guides/orchestration/airflow-and-dbt-cloud/connection-type-configured.png) @@ -145,7 +145,7 @@ Now you have all the working pieces to get up and running with Airflow + dbt Clo Both IDs are included inside of the dbt Cloud job URL as shown in the following snippets: ```python -# For the dbt Cloud Job URL https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/ +# For the dbt Cloud Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The account_id is 16173 # Update line 28 @@ -153,7 +153,7 @@ default_args={"dbt_cloud_conn_id": "dbt_cloud", "account_id": 16173}, ``` ```python -# For the dbt Cloud Job URL https://cloud.getdbt.com/#/accounts/16173/projects/36467/jobs/65767/ +# For the dbt Cloud Job URL https://YOUR_ACCESS_URL/#/accounts/16173/projects/36467/jobs/65767/ # The job_id is 65767 # Update line 39 diff --git a/website/docs/guides/bigquery-qs.md b/website/docs/guides/bigquery-qs.md index 1ba5f7b0021..e608efeffc7 100644 --- a/website/docs/guides/bigquery-qs.md +++ b/website/docs/guides/bigquery-qs.md @@ -85,7 +85,7 @@ In order to let dbt connect to your warehouse, you'll need to generate a keyfile 3. Create a service account key for your new project from the [Service accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts?walkthrough_id=iam--create-service-account-keys&start_index=1#step_index=1). For more information, refer to [Create a service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating) in the Google Cloud docs. When downloading the JSON file, make sure to use a filename you can easily remember. For example, `dbt-user-creds.json`. For security reasons, dbt Labs recommends that you protect this JSON file like you would your identity credentials; for example, don't check the JSON file into your version control software. ## Connect dbt Cloud to BigQuery​ -1. Create a new project in [dbt Cloud](https://cloud.getdbt.com/). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. +1. Create a new project in [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. 2. Enter a project name and click **Continue**. 3. For the warehouse, click **BigQuery** then **Next** to set up your connection. 4. Click **Upload a Service Account JSON File** in settings. diff --git a/website/docs/guides/core-cloud-2.md b/website/docs/guides/core-cloud-2.md index 335b164d988..93e9e92bfa4 100644 --- a/website/docs/guides/core-cloud-2.md +++ b/website/docs/guides/core-cloud-2.md @@ -35,7 +35,7 @@ The guide outlines the following steps: ## Considerations -If your team has is using dbt Core today, you could be reading this guide because: +If your team is using dbt Core today, you could be reading this guide because: - You’ve realized the burden of maintaining that deployment. - The person who set it up has since left. - You’re interested in what dbt Cloud could do to better manage the complexity of your dbt deployment, democratize access to more contributors, or improve security and governance practices. @@ -46,13 +46,13 @@ The most important things you need to think about when moving from dbt Core to d - How is your team structured? Are there natural divisions of domain? - Should you have one project or multiple? Which dbt resources do you want to standardize & keep central? -- Who should have permissions to view, develop, administer? +- Who should have permission to view, develop, and administer? - How are you scheduling your dbt models to run in production? - How are you currently managing Continuous integration/Continuous deployment (CI/CD) of logical changes (if at all)? - How do your data developers prefer to work? - How do you manage different data environments and the different behaviors in those environments? -dbt Cloud provides standard mechanisms for tackling these considerations, all of which delivers long-term benefits to your organization: +dbt Cloud provides standard mechanisms for tackling these considerations, all of which deliver long-term benefits to your organization: - Cross-team collaboration - Access control - Orchestration @@ -141,7 +141,7 @@ After [setting the foundations of dbt Cloud](https://docs.getdbt.com/guides/core Once you’ve confirmed that dbt Cloud orchestration and CI/CD are working as expected, you should pause your current orchestration tool and stop or update your current CI/CD process. This is not relevant if you’re still using an external orchestrator (such as Airflow), and you’ve swapped out `dbt-core` execution for dbt Cloud execution (through the [API](/docs/dbt-cloud-apis/overview)). Familiarize your team with dbt Cloud's [features](/docs/cloud/about-cloud/dbt-cloud-features) and optimize development and deployment processes. Some key features to consider include: -- **Version management:** Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, removing the hassle of manual updates and version discrepancies. You can go versionless by opting to **[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)** to always get the latest features and early access to new functionality for your dbt project. +- **Version management:** Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, removing the hassle of manual updates and version discrepancies. You can go [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest features and early access to new functionality for your dbt project. - **Development tools**: Use the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) or [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud) to build, test, run, and version control your dbt projects. - **Documentation and Source freshness:** Automate storage of [documentation](/docs/build/documentation) and track [source freshness](/docs/deploy/source-freshness) in dbt Cloud, which streamlines project maintenance. - **Notifications and logs:** Receive immediate [notifications](/docs/deploy/monitor-jobs) for job failures, with direct links to the job details. Access comprehensive logs for all job runs to help with troubleshooting. diff --git a/website/docs/guides/core-to-cloud-1.md b/website/docs/guides/core-to-cloud-1.md index 6e130d3a29f..171e844d7e5 100644 --- a/website/docs/guides/core-to-cloud-1.md +++ b/website/docs/guides/core-to-cloud-1.md @@ -57,7 +57,7 @@ This guide outlines the steps you need to take to move from dbt Core to dbt Clou ## Prerequisites - You have an existing dbt Core project connected to a Git repository and data platform supported in [dbt Cloud](/docs/cloud/connect-data-platform/about-connections). -- A [supported version](/docs/dbt-versions/core) of dbt or select [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) of dbt. +- A [supported version](/docs/dbt-versions/core) of dbt or select [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) of dbt. - You have a dbt Cloud account. **[Don't have one? Start your free trial today](https://www.getdbt.com/signup)**! ## Account setup @@ -144,7 +144,7 @@ The most common data environments are production, staging, and development. The 1. **Set up development environment** — Set up your [development](/docs/dbt-cloud-environments#create-a-development-environment) environment and [development credentials](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#access-the-cloud-ide). You’ll need this to access your dbt project and start developing. 2. **dbt Core version** — In your dbt Cloud environment and credentials, use the same dbt Core version you use locally. You can run `dbt --version` in the command line to find out which version of dbt Core you’re using. - - When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) so you don’t have to. + - When using dbt Core, you need to think about which version you’re using and manage your own upgrades. When using dbt Cloud, leverage ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) so you don’t have to. 3. **Connect to your data platform** — When using dbt Cloud, you can [connect to your data platform](/docs/cloud/connect-data-platform/about-connections) directly in the UI. - Each environment is roughly equivalent to an entry in your `profiles.yml` file. This means you don't need a `profiles.yml` file in your project. @@ -171,9 +171,9 @@ In dbt Cloud, you can set [environment variables](/docs/build/environment-variab In dbt Core, environment variables, or the [`env_var` function](/reference/dbt-jinja-functions/env_var), are defined manually by the developer or within the external application running dbt. ### Environment variables in dbt Cloud - - dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET_``DBT_ENV_SECRET`). + - dbt Cloud environment variables must be prefixed with `DBT_` (including `DBT_ENV_CUSTOM_ENV_` or `DBT_ENV_SECRET`). - If your dbt Core environment variables don’t follow this naming convention, perform a [“find and replace”](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud#dbt-cloud-ide-features) in your project to make sure all references to these environment variables contain the proper naming conventions. -- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET_``DBT_ENV_SECRET` to obscure them in logs and the UI. +- dbt Cloud secures environment variables that enable more flexible configuration of data warehouse connections or git provider integrations, offering additional measures for sensitive values, such as prefixing keys with `DBT_ENV_SECRET`to obscure them in logs and the UI. @@ -206,7 +206,7 @@ To use the [dbt Cloud's job scheduler](/docs/deploy/job-scheduler), set up one e ### Initial setup steps 1. **dbt Core version** — In your environment settings, configure dbt Cloud with the same dbt Core version. - - Once your full migration is complete, we recommend upgrading your environments to a versionless experience by opting to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) to always get the latest features and more. You only need to do this once. + - Once your full migration is complete, we recommend upgrading your environments to ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest features and more. You only need to do this once. 2. **Configure your jobs** — [Create jobs](/docs/deploy/deploy-jobs#create-and-schedule-jobs) for scheduled or event-driven dbt jobs. You can use cron execution, manual, pull requests, or trigger on the completion of another job. - Note that alongside [jobs in dbt Cloud](/docs/deploy/jobs), discover other ways to schedule and run your dbt jobs with the help of other tools. Refer to [Integrate with other tools](/docs/deploy/deployment-tools) for more information. diff --git a/website/docs/guides/core-to-cloud-3.md b/website/docs/guides/core-to-cloud-3.md index cf3cd58d0d1..0ea22de8478 100644 --- a/website/docs/guides/core-to-cloud-3.md +++ b/website/docs/guides/core-to-cloud-3.md @@ -36,7 +36,7 @@ You may have already started your move to dbt Cloud and are looking for tips to In dbt Cloud, you can natively connect to your data platform and test its [connection](/docs/connect-adapters) with a click of a button. This is especially useful for users who are new to dbt Cloud or are looking to streamline their connection setup. Here are some tips and caveats to consider: ### Tips -- Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, eliminating the need for manual updates and version discrepancies. You can go versionless and **[Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version)** to always get the latest features and early access to new functionality for your dbt project. +- Manage [dbt versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) and ensure team collaboration with dbt Cloud's one-click feature, eliminating the need for manual updates and version discrepancies. You can go [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest features and early access to new functionality for your dbt project. - dbt Cloud supports a whole host of [cloud providers](/docs/cloud/connect-data-platform/about-connections), including Snowflake, Databricks, BigQuery, Fabric, and Redshift (to name a few). - Use [Extended Attributes](/docs/deploy/deploy-environments#extended-attributes) to set a flexible [profiles.yml](/docs/core/connect-data-platform/profiles.yml) snippet in your dbt Cloud environment settings. It gives you more control over environments (both deployment and development) and extends how dbt Cloud connects to the data platform within a given environment. - For example, if you have a field in your `profiles.yml` that you’d like to add to the dbt Cloud adapter user interface, you can use Extended Attributes to set it. diff --git a/website/docs/guides/customize-schema-alias.md b/website/docs/guides/customize-schema-alias.md new file mode 100644 index 00000000000..28d4aada525 --- /dev/null +++ b/website/docs/guides/customize-schema-alias.md @@ -0,0 +1,545 @@ +--- +title: Customize dbt models database, schema, and alias +id: customize-schema-alias +description: "Learn how to properly adjust your generate_schema_name() and generate_alias_name() macros." +displayText: Learn how to adjust your generate schema name and generate alias name. +hoverSnippet: Learn how to adjust your generate schema name and generate alias name. +# time_to_complete: '30 minutes' commenting out until we test +icon: 'guides' +hide_table_of_contents: true +level: 'Advanced' +recently_updated: true +--- + +
+ +## Introduction +This guide explains how to customize the [schema](/docs/build/custom-schemas), [database](/docs/build/custom-databases), and [alias](/docs/build/custom-aliases) naming conventions in dbt to fit your data warehouse governance and design needs. +When we develop dbt models and execute certain [commands](https://docs.getdbt.com/reference/dbt-commands) (such as `dbt run` or `dbt build`), objects (like tables and views) get created in the data warehouse based on these naming conventions. + + + +:::info A word on naming + +Different warehouses have different names for _logical databases_. The information in this document covers "databases" on Snowflake, Redshift, and Postgres; "projects" on BigQuery; and "catalogs" on Databricks Unity Catalog. + +::: + + +The following is dbt's out-of-the-box default behavior: + +- The database where the object is created is defined by the database configured at the [environment level in dbt Cloud](/docs/dbt-cloud-environments) or in the [`profiles.yml` file](/docs/core/connect-data-platform/profiles.yml) in dbt Core. + +- The schema depends on whether you have defined a [custom schema](/docs/build/custom-schemas) for the model: + - If you haven't defined a custom schema, dbt creates the object in the default schema. In dbt Cloud this is typically `dbt_username` for development and the default schema for deployment environments. In dbt Core, it uses the schema specified in the `profiles.yml` file. + - If you define a custom schema, dbt concatenates the schema mentioned earlier with the custom one. + - For example, if the configured schema is `dbt_myschema` and the custom one is `marketing`, the objects will be created under `dbt_myschema_marketing`. + - Note that for automated CI jobs, the schema name derives from the job number and PR number: `dbt_cloud_pr__`. + + +- The object name depends on whether an [alias](/reference/resource-configs/alias) has been defined on the model: + - If no alias is defined, the object will be created with the same name as the model, without the `.sql` or `.py` at the end. + - For example, suppose that we have a model where the sql file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and no custom alias is configured. The resulting model will be created in `dbt_myschema_marketing.fct_orders_complete` in the dev environment. + - If an alias is defined, the object will be created with the configured alias. + - For example, suppose that we have a model where the sql file is titled `fct_orders_complete.sql`, the custom schema is `marketing`, and the alias is configured to be `fct_orders`. The resulting model will be created in `dbt_myschema_marketing.fct_orders` + +These default rules are a great starting point, and many organizations choose to stick with those without any customization required. + +The defaults allow developers to work in their isolated schemas (sandboxes) without overwriting each other's work — even if they're working on the same tables. + + +## How to customize this behavior + +While the default behavior will fit the needs of most organizations, there are occasions where this approach won't work. + +For example, dbt expects that it has permission to create schemas as needed (and we recommend that the users running dbt have this ability), but it might not be allowed at your company. + +Or, based on how you've designed your warehouse, you may wish to minimize the number of schemas in your dev environment (and avoid schema sprawl by not creating the combination of all developer schemas and custom schemas). + +Alternatively, you may even want your dev schemas to be named after feature branches instead of the developer name. + +For this reason, dbt offers three macros to customize what objects are created in the data warehouse: + +- [`generate_database_name()`](/docs/build/custom-databases#generate_database_name) +- [`generate_schema_name()`](/docs/build/custom-schemas#how-does-dbt-generate-a-models-schema-name) +- [`generate_alias_name()`](/docs/build/custom-aliases#generate_alias_name) + +By overwriting one or multiple of those macros, we can tailor where dbt objects are created in the data warehouse and align with any existing requirement. + + +:::note Key concept + +Models run from two different contexts must result in unique objects in the data warehouse. For example, a developer named Suzie is working on enhancements to `fct_player_stats`, but Darren is developing against the exact same object. + +In order to prevent overwriting each other's work, both Suzie and Darren should each have their unique versions of `fct_player_stats` in the development environment. + +Further, the staging version of `fct_player_stats` should exist in a unique location apart from the development versions, and the production version. + +::: + + +We often leverage the following when customizing these macros: + +- In dbt Cloud, we recommend utilizing [environment variables](/docs/build/environment-variables) to define where the dbt invocation is occurring (dev/stg/prod). + - They can be set at the environment level and all jobs will automatically inherit the default values. We'll add jinja logic (`if/else/endif`) to identify whether the run happens in dev, prod, Ci, and more. + +- Or as an alternative to environment variables, you can use `target.name`. For more information, you can refer to [About target variables](/reference/dbt-jinja-functions/target). + + + + +To allow the database/schema/object name to depend on the current branch, you can use the out of the box `DBT_CLOUD_GIT_BRANCH` environment variable in dbt Cloud [special environment variables](/docs/build/environment-variables#special-environment-variables). + + +## Example use cases + +Here are some typical examples we've encountered with dbt users leveraging those 3 macros and different logic. + + +:::note + +Note that the following examples are not comprehensive and do not cover all the available options. These examples are meant to be templates for you to develop your own behaviors. + +::: + + +- [Use custom schema without concatenating target schema in production](/guides/customize-schema-alias?step=3#1-custom-schemas-without-target-schema-concatenation-in-production) +- [Add developer identities to tables](/guides/customize-schema-alias?step=3#2-static-schemas-add-developer-identities-to-tables) +- [Use branch name as schema prefix](/guides/customize-schema-alias?step=3#3-use-branch-name-as-schema-prefix) +- [Use a static schema for CI](/guides/customize-schema-alias?step=3#4-use-a-static-schema-for-ci) + + +### 1. Custom schemas without target schema concatenation in production + + +The most common use case is using the custom schema without concatenating it with the default schema name when in production. + +To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: + + + + + +```jinja +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- elif env_var('DBT_ENV_TYPE','DEV') == 'PROD' -%} + + {{ custom_schema_name | trim }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Developer 1 | dev | dbt_dev1 |dev.dbt_dev1_marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.dbt_dev2_marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model| +| CI PR 234 | ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model| +| Production | prod | analytics |prod.marketing.my_model | + + +:::note + +We added logic to check if the current dbt run is happening in production or not. This is important, and we explain why in the [What not to do](/guides/customize-schema-alias?step=3#what-not-to-do) section. + +::: + + +### 2. Static schemas: Add developer identities to tables + +Occasionally, we run into instances where the security posture of the organization prevents developers from creating schemas and all developers have to develop in a single schema. + +In this case, we can: + +- Create a new file called generate_schema_name.sql under your macros folder with the following code: + +- Change `generate_schema_name()` to use a single schema for all developers, even if a custom schema is set. +- Update `generate_alias_name()` to append the developer alias and the custom schema to the front of the table name in the dev environment. + - This method is not ideal, as it can cause long table names, but it will let developers see in which schema the model will be created in production. + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- elif env_var('DBT_ENV_TYPE','DEV') != 'CI' -%} + + {{ custom_schema_name | trim }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + + + +```jinja + +{% macro generate_alias_name(custom_alias_name=none, node=none) -%} + + {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} + + {%- if custom_alias_name -%} + + {{ target.schema }}__{{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ target.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} + + {%- else -%} + + {{ target.schema }}__{{ node.name }} + + {%- endif -%} + + {%- else -%} + + {%- if custom_alias_name -%} + + {{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} + + {%- else -%} + + {{ node.name }} + + {%- endif -%} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Developer 1 | dev | dbt_dev1 |dev.marketing.dbt_dev1_my_model | +| Developer 2 | dev | dbt_dev2 |dev.marketing.dbt_dev2_my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model| +| CI PR 234 | ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model| +| Production | prod | analytics |prod.marketing.my_model | + + +### 3. Use branch name as schema prefix + +For teams who prefer to isolate work based on the feature branch, you may want to take advantage of the `DBT_CLOUD_GIT_BRANCH` special environment variable. Please note that developers will write to the exact same schema when they are on the same feature branch. + + +:::note + +The `DBT_CLOUD_GIT_BRANCH` variable is only available within the dbt Cloud IDE and not the Cloud CLI. + +::: + + +We’ve also seen some organizations prefer to organize their dev databases by branch name. This requires implementing similar logic in `generate_database_name()` instead of the `generate_schema_name()` macro. By default, dbt will not automatically create the databases. + +Refer to the [Tips and tricks](https://docs.getdbt.com/guides/customize-schema-alias?step=5) section to learn more. + + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if env_var('DBT_ENV_TYPE','DEV') == 'DEV' -%} + + {#- we replace characters not allowed in the schema names by "_" -#} + {%- set re = modules.re -%} + {%- set cleaned_branch = re.sub("\W", "_", env_var('DBT_CLOUD_GIT_BRANCH')) -%} + + {%- if custom_schema_name is none -%} + + {{ cleaned_branch }} + + {%- else -%} + + {{ cleaned_branch }}_{{ custom_schema_name | trim }} + + {%- endif -%} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Branch |Target database| Target schema | Resulting object | +|-------------|:----------:|:-------------:|:-------------:|:---------------------------------:| +| Developer 1 |`featureABC`|dev | dbt_dev1 |dev.featureABC_marketing.my_model | +| Developer 2 |`featureABC`|dev | dbt_dev2 |dev.featureABC_marketing.my_model | +| Developer 1 |`feature123`|dev | dbt_dev1 |dev.feature123_marketing.my_model | +| CI PR 123 | |ci | dbt_pr_123 |ci.dbt_pr_123_marketing.my_model | +| CI PR 234 | |ci | dbt_pr_234 |ci.dbt_pr_234_marketing.my_model | +| Production | |prod | analytics |prod.marketing.my_model | + + +When developer 1 and developer 2 are checked out on the same branch, they will generate the same object in the data warehouse. This shouldn't be a problem as being on the same branch means the model's code will be the same for both developers. + + +### 4. Use a static schema for CI + +Some organizations prefer to write their CI jobs to a single schema with the PR identifier prefixed to the front of the table name. It's important to note that this will result in long table names. + +To do so, you can create a new file called `generate_schema_name.sql` under your macros folder with the following code: + + + + +```jinja + +{% macro generate_schema_name(custom_schema_name=none, node=none) -%} + + {%- set default_schema = target.schema -%} + + {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} + {# {%- if target.name == 'CI' -%} #} + + {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} + + ci_schema + + {%- elif custom_schema_name is none -%} + + {{ default_schema }} + + {%- else -%} + + {{ default_schema }}_{{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + + + +```jinja + +{% macro generate_alias_name(custom_alias_name=none, node=none) -%} + + {# If the CI Job does not exist in its own environment, use the target.name variable inside the job instead #} + {# {%- if target.name == 'CI' -%} #} + {%- if env_var('DBT_ENV_TYPE','DEV') == 'CI' -%} + + {%- if custom_alias_name -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ node.name ~ "_v" ~ (node.version | replace(".", "_")) }} + + {%- else -%} + + {{ target.schema }}__{{ node.config.schema }}__{{ node.name }} + + {%- endif -%} + + {%- else -%} + + {%- if custom_alias_name -%} + + {{ custom_alias_name | trim }} + + {%- elif node.version -%} + + {{ return(node.name ~ "_v" ~ (node.version | replace(".", "_"))) }} + + {%- else -%} + + {{ node.name }} + + {%- endif -%} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +This will generate the following outputs for a model called `my_model` with a custom schema of `marketing`, preventing any overlap of objects between dbt runs from different contexts. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:----------------------------------------: | +| Developer 1 | dev | dbt_dev1 |dev.dbt_dev1_marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.dbt_dev2_marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.ci_schema.dbt_pr_123_marketing_my_model | +| CI PR 234 | ci | dbt_pr_234 |ci.ci_schema.dbt_pr_234_marketing_my_model | +| Production | prod | analytics |prod.marketing.my_model | + + +## What not to do + +This section will provide an outline of what users should avoid doing when customizing their schema and alias due to the issues that may arise. + + +### Update generate_schema_name() to always use the custom schema + + +Some people prefer to only use the custom schema when it is set instead of concatenating the default schema with the custom one, as it happens in the out of the box behavior. + + +### Problem + +When modifying the default macro for `generate_schema_name()`, this might result in creating this new version. + + + +```jinja + +{% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none -%} + + {{ default_schema }} + + {%- else -%} + # The following is incorrect as it omits {{ default_schema }} before {{ custom_schema_name | trim }}. + {{ custom_schema_name | trim }} + + {%- endif -%} + +{%- endmacro %} + +``` + + + +While it may provide the expected output for production, where a dedicated database is used, it will generate conflicts anywhere people share a database. + + +Let’s look at the example of a model called `my_model` with a custom schema of `marketing`. + + +| Context |Target database| Target schema | Resulting object | +|-------------|:-------------:|:-------------:|:------------------------------:| +| Production | prod | analytics |prod.marketing.my_model | +| Developer 1 | dev | dbt_dev1 |dev.marketing.my_model | +| Developer 2 | dev | dbt_dev2 |dev.marketing.my_model | +| CI PR 123 | ci | dbt_pr_123 |ci.marketing.my_model | +| CI PR 234 | ci | dbt_pr_234 |ci.marketing.my_model | + + + +We can see that both developer 1 and developer 2 get the same object for `my_model`. This means that if they both work on this model at the same time, it will be impossible to know if the version currently in the data warehouse is the one from developer 1 and developer 2. + +Similarly, different PRs will result in the exact same object in the data warehouse. If different PRs are open at the same time and modifying the same models, it is very likely that we will get issues, slowing down the whole development and code promotion. + + +### Solution + +As described in the previous example, update the macro to check if dbt is running in production. Only in production should we remove the concatenation and use the custom schema alone. + + +## Tips and tricks + +This section will provide some useful tips on how to properly adjust your `generate_database_name()` and `generate_alias_name()` macros. + + +### Creating non existing databases from dbt + +dbt will automatically try to create a schema if it doesn’t exist and if an object needs to be created in it, but it won’t automatically try to create a database that doesn’t exist. + +So, if your `generate_database_name()` configuration points to different databases, which might not exist, dbt will fail if you do a simple `dbt build`. + +It is still possible to get it working in dbt by creating some macros that will check if a database exists and if not, dbt will create it. You can then call those macros either in [a `dbt run-operation ...` step](/reference/commands/run-operation) in your jobs or as a [`on-run-start` hook](/reference/project-configs/on-run-start-on-run-end). + + +### Assuming context using environment variables rather than `target.name` + + +We prefer to use [environment variables](/docs/build/environment-variables) over `target.name` For a further read, have a look at ([About target variables](/reference/dbt-jinja-functions/target)) to decipher the context of the dbt invocation. + +- `target.name` cannot be set at the environment-level. Therefore, every job within the environment must explicitly specify the `target.name` override. If the job does not have the appropriate `target.name` value set, the database/schema/alias may not resolve properly. Alternatively, environment variable values are inherited by the jobs within their corresponding environment. The environment variable values can also be overwritten within the jobs if needed. + + + + + +- `target.name` requires every developer to input the same value (often ‘dev’) into the target name section of their project development credentials. If a developer doesn’t have the appropriate target name value set, their database/schema/alias may not resolve properly. + + + + + +### Always enforce custom schemas + +Some users prefer to enforce custom schemas on all objects within their projects. This avoids writing to unintended “default” locations. You can add this logic to your `generate_schema_name()` macro to [raise a compilation error](/reference/dbt-jinja-functions/exceptions) if a custom schema is not defined for an object. + + + + +```jinja + + {% macro generate_schema_name(custom_schema_name, node) -%} + + {%- set default_schema = target.schema -%} + {%- if custom_schema_name is none and node.resource_type == 'model' -%} + + {{ exceptions.raise_compiler_error("Error: No Custom Schema Defined for the model " ~ node.name ) }} + + {%- endif -%} + +``` + + +
diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index b969786b384..bb248e09320 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -169,16 +169,63 @@ If you get a session error and don’t get redirected to this page, you can go b There are two ways to connect dbt Cloud to Databricks. The first option is Partner Connect, which provides a streamlined setup to create your dbt Cloud account from within your new Databricks trial account. The second option is to create your dbt Cloud account separately and build the Databricks connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt Cloud setup flow, dbt Labs recommends connecting manually. -If you want to use Partner Connect, refer to [Connect to dbt Cloud using Partner Connect](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-using-partner-connect) in the Databricks docs for instructions. +## Set up the integration from Partner Connect -If you want to connect manually, refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. +:::note + Partner Connect is intended for trial partner accounts. If your organization already has a dbt Cloud account, connect manually. Refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. +::: + +To connect dbt Cloud to Databricks using Partner Connect, do the following: + +1. In the sidebar of your Databricks account, click **Partner Connect**. + +2. Click the **dbt tile**. + +3. Select a catalog from the drop-down list, and then click **Next**. The drop-down list displays catalogs you have read and write access to. If your workspace isn't `-enabled`, the legacy Hive metastore (`hive_metastore`) is used. + +5. If there are SQL warehouses in your workspace, select a SQL warehouse from the drop-down list. If your SQL warehouse is stopped, click **Start**. + +6. If there are no SQL warehouses in your workspace: + + 1. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. + 2. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. + 3. Return to the Partner Connect tab in your browser, and then close the **dbt tile**. + 4. Re-open the **dbt tile**. + 5. Select the SQL warehouse you just created from the drop-down list. + +7. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you have read and write access to. You can repeat this step to add multiple schemas. -## Set up a dbt Cloud managed repository -If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as the Partner Connect provides you with a managed repository. Otherwise, you will need to create your repository connection. + Partner Connect creates the following resources in your workspace: + + - A Databricks service principal named **DBT_CLOUD_USER**. + - A Databricks personal access token that is associated with the **DBT_CLOUD_USER** service principal. + + Partner Connect also grants the following privileges to the **DBT_CLOUD_USER** service principal: + + - (Unity Catalog) **USE CATALOG**: Required to interact with objects within the selected catalog. + - (Unity Catalog) **USE SCHEMA**: Required to interact with objects within the selected schema. + - (Unity Catalog) **CREATE SCHEMA**: Grants the ability to create schemas in the selected catalog. + - (Hive metastore) **USAGE**: Required to grant the **SELECT** and **READ_METADATA** privileges for the schemas you selected. + - **SELECT**: Grants the ability to read the schemas you selected. + - (Hive metastore) **READ_METADATA**: Grants the ability to read metadata for the schemas you selected. + - **CAN_USE**: Grants permissions to use the SQL warehouse you selected. + +8. Click **Next**. + + The **Email** box displays the email address for your Databricks account. dbt Labs uses this email address to prompt you to create a trial dbt Cloud account. + +9. Click **Connect to dbt Cloud**. + + A new tab opens in your web browser, which displays the getdbt.com website. + +10. Complete the on-screen instructions on the getdbt.com website to create your trial dbt Cloud account. + +## Set up a dbt Cloud managed repository ## Initialize your dbt project​ and start developing + Now that you have a repository configured, you can initialize your project and start development in dbt Cloud: 1. Click **Start developing in the IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. diff --git a/website/docs/guides/debug-schema-names.md b/website/docs/guides/debug-schema-names.md index f73ba30fa5a..10ead21a88f 100644 --- a/website/docs/guides/debug-schema-names.md +++ b/website/docs/guides/debug-schema-names.md @@ -93,7 +93,7 @@ Now, re-read through the logic of your `generate_schema_name` macro, and mentall You should find that the schema dbt is constructing for your model matches the output of your `generate_schema_name` macro. -Be careful. Snapshots do not follow this behavior, check out the docs on [target_schema](/reference/resource-configs/target_schema) instead. +Be careful. Snapshots do not follow this behavior if target_schema is set. To have environment-aware snapshots in v1.9+ or dbt Cloud, remove the [target_schema config](/reference/resource-configs/target_schema) from your snapshots. If you still want a custom schema for your snapshots, use the [`schema`](/reference/resource-configs/schema) config instead. ## Adjust as necessary @@ -103,4 +103,4 @@ Now that you understand how a model's schema is being generated, you can adjust If you change the logic in `generate_schema_name`, it's important that you consider whether two users will end up writing to the same schema when developing dbt models. This consideration is the reason why the default implementation of the macro concatenates your target schema and custom schema together — we promise we were trying to be helpful by implementing this behavior, but acknowledge that the resulting schema name is unintuitive. - \ No newline at end of file + diff --git a/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md b/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md index b4cea114f1a..f420b7845a2 100644 --- a/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md +++ b/website/docs/guides/how-to-use-databricks-workflows-to-run-dbt-cloud-jobs.md @@ -130,7 +130,7 @@ if __name__ == '__main__': 4. Replace **``** and **``** with the correct values of your environment and [Access URL](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan. - * To find these values, navigate to **dbt Cloud**, select **Deploy -> Jobs**. Select the Job you want to run and copy the URL. For example: `https://cloud.getdbt.com/deploy/000000/projects/111111/jobs/222222` + * To find these values, navigate to **dbt Cloud**, select **Deploy -> Jobs**. Select the Job you want to run and copy the URL. For example: `https://YOUR_ACCESS_URL/deploy/000000/projects/111111/jobs/222222` and therefore valid code would be: Your URL is structured `https:///deploy//projects//jobs/` diff --git a/website/docs/guides/mesh-qs.md b/website/docs/guides/mesh-qs.md index fe581471da3..f2e02437819 100644 --- a/website/docs/guides/mesh-qs.md +++ b/website/docs/guides/mesh-qs.md @@ -40,7 +40,7 @@ To leverage dbt Mesh, you need the following: - You must have a [dbt Cloud Enterprise account](https://www.getdbt.com/get-started/enterprise-contact-pricing) - You have access to a cloud data platform, permissions to load the sample data tables, and dbt Cloud permissions to create new projects. -- Set your development and deployment [environments](/docs/dbt-cloud-environments) to use dbt [version](/docs/dbt-versions/core) 1.6 or later. You can also opt to go versionless and select [Keep on latest version of](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) to always get the most recent features and functionality. +- Set your development and deployment [environments](/docs/dbt-cloud-environments) to use dbt [version](/docs/dbt-versions/core) 1.6 or later. You can also opt to go ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the most recent features and functionality. - This guide uses the Jaffle Shop sample data, including `customers`, `orders`, and `payments` tables. Follow the provided instructions to load this data into your respective data platform: - [Snowflake](https://docs.getdbt.com/guides/snowflake?step=3) - [Databricks](https://docs.getdbt.com/guides/databricks?step=3) diff --git a/website/docs/guides/redshift-qs.md b/website/docs/guides/redshift-qs.md index e3685595804..544c18a75d5 100644 --- a/website/docs/guides/redshift-qs.md +++ b/website/docs/guides/redshift-qs.md @@ -166,7 +166,7 @@ Now we are going to load our sample data into the S3 bucket that our Cloudformat select * from stripe.payment; ``` ## Connect dbt Cloud to Redshift -1. Create a new project in [dbt Cloud](https://cloud.getdbt.com/). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. +1. Create a new project in [dbt Cloud](/docs/cloud/about-cloud/access-regions-ip-addresses). From **Account settings** (using the gear menu in the top right corner), click **+ New Project**. 2. Enter a project name and click **Continue**. 3. For the warehouse, click **Redshift** then **Next** to set up your connection. 4. Enter your Redshift settings. Reference your credentials you saved from the CloudFormation template. diff --git a/website/docs/guides/sl-snowflake-qs.md b/website/docs/guides/sl-snowflake-qs.md index 9fb42fe1828..6226634fe42 100644 --- a/website/docs/guides/sl-snowflake-qs.md +++ b/website/docs/guides/sl-snowflake-qs.md @@ -22,14 +22,6 @@ import ConnectQueryAPI from '/snippets/_sl-connect-and-query-api.md'; import RunProdJob from '/snippets/_sl-run-prod-job.md'; import SlSetUp from '/snippets/_new-sl-setup.md'; - - -import DeprecationNotice from '/snippets/_sl-deprecation-notice.md'; - - - - - ## Introduction The [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), powered by [MetricFlow](/docs/build/about-metricflow), simplifies the setup of key business metrics. It centralizes definitions, avoids duplicate code, and ensures easy access to metrics in downstream tools. MetricFlow helps manage company metrics easier, allowing you to define metrics in your dbt project and query them in dbt Cloud with [MetricFlow commands](/docs/build/metricflow-commands). @@ -41,9 +33,9 @@ import SLCourses from '/snippets/_sl-course.md'; This quickstart guide is designed for dbt Cloud users using Snowflake as their data platform. It focuses on building and defining metrics, setting up the dbt Semantic Layer in a dbt Cloud project, and querying metrics in Google Sheets. -**For users on different data platform** +**For users on different data platforms** -If you're using a data platform other than Snowflake, this guide is also be applicable to you. You can adapt the setup for your specific platform by following the account setup and data loading instructions detailed in the following tabs for each respective platform. +If you're using a data platform other than Snowflake, this guide is also applicable to you. You can adapt the setup for your specific platform by following the account setup and data loading instructions detailed in the following tabs for each respective platform. The rest of this guide applies universally across all supported platforms, ensuring you can fully leverage the dbt Semantic Layer. @@ -110,11 +102,11 @@ Open a new tab and follow these quick steps for account setup and data loading i - Enterprise — Developer license with Account Admin permissions. Or "Owner" with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. - Team — "Owner" access with a Developer license. - - Trial — Automatic "Owner" access under a trail of the Team plan. + - Trial — Automatic "Owner" access under a Team plan trial. -- Production and development environments must be on [dbt version 1.6 or higher](/docs/dbt-versions/upgrade-dbt-version-in-cloud). Alternatively, set your environment to "versionless" by selecting [ Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) to always get the latest updates. +- Production and development environments must be on [dbt version 1.6 or higher](/docs/dbt-versions/upgrade-dbt-version-in-cloud). Alternatively, set your environment to [**Versionless**](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) to always get the latest updates. - Create a [trial Snowflake account](https://signup.snowflake.com/): - Select the Enterprise Snowflake edition with ACCOUNTADMIN access. Consider organizational questions when choosing a cloud provider, refer to Snowflake's [Introduction to Cloud Platforms](https://docs.snowflake.com/en/user-guide/intro-cloud-platforms). - Select a cloud provider and region. All cloud providers and regions will work so choose whichever you prefer. @@ -128,7 +120,7 @@ This guide will cover the following topics: - [Load sample data into your Snowflake account](/guides/sl-snowflake-qs?step=4) - [Connect dbt Cloud to Snowflake](/guides/sl-snowflake-qs?step=5) - [Set up a dbt Cloud managed repository](/guides/sl-snowflake-qs?step=6) -- [Initialized a dbt Cloud project and start developer](/guides/sl-snowflake-qs?step=7) +- [Initialize a dbt Cloud project and start developing](/guides/sl-snowflake-qs?step=7) - [Build your dbt Cloud project](/guides/sl-snowflake-qs?step=8) - [Create a semantic model in dbt Cloud](/guides/sl-snowflake-qs?step=9) - [Define metrics in dbt Cloud](/guides/sl-snowflake-qs?step=10) @@ -281,7 +273,7 @@ Using Partner Connect allows you to create a complete dbt account with your [Sno -2. In the **Connect to dbt** popup, find the **Optional Grant** option and select the **RAW** and **ANALYTICS** databases. This will grant access for your new dbt user role to each database. Then, click **Connect**. +2. In the **Connect to dbt** popup, find the **Optional Grant** option and select the **RAW** and **ANALYTICS** databases. This will grant access for your new dbt user role to each selected database. Then, click **Connect**. @@ -293,13 +285,13 @@ Using Partner Connect allows you to create a complete dbt account with your [Sno -4. After the new tab loads, you will see a form. If you already created a dbt Cloud account, you will be asked to provide an account name. If you haven't created account, you will be asked to provide an account name and password. +4. After the new tab loads, you will see a form. If you already created a dbt Cloud account, you will be asked to provide an account name. If you haven't created an account, you will be asked to provide an account name and password. 5. After you have filled out the form and clicked **Complete Registration**, you will be logged into dbt Cloud automatically. -6. From your **Account Settings** in dbt Cloud (using the gear menu in the upper right corner), choose the "Partner Connect Trial" project and select **snowflake** in the overview table. Select edit and update the fields **Database** and **Warehouse** to be `analytics` and `transforming`, respectively. +6. From your **Account Settings** in dbt Cloud (using the gear menu in the upper right corner), choose the "Partner Connect Trial" project and select **snowflake** in the overview table. Select **Edit** and update the **Database** field to `analytics` and the **Warehouse** field to `transforming`. @@ -342,7 +334,7 @@ Using Partner Connect allows you to create a complete dbt account with your [Sno ## Set up a dbt Cloud managed repository -If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as the Partner Connect provides you with a managed repository. Otherwise, you will need to create your repository connection. +If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as Partner Connect provides you with a [managed repository](/docs/collaborate/git/managed-repository). Otherwise, you will need to create your repository connection. @@ -355,7 +347,7 @@ Now that you have a repository configured, you can initialize your project and s 1. Click **Start developing in the dbt Cloud IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. 2. Above the file tree to the left, click **Initialize your project**. This builds out your folder structure with example models. -3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit`. This creates the first commit to your managed repo and allows you to open a branch where you can add new dbt code. +3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit`. This creates the first commit to your managed repo and allows you to open a branch where you can add a new dbt code. 4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now: - Delete the models/examples folder in the **File Explorer**. - Click **+ Create new file**, add this query to the new file, and click **Save as** to save the new file: @@ -378,7 +370,7 @@ You have two options for working with files in the dbt Cloud IDE: Name the new branch `build-project`. -1. Hover over the `models` directory and click the three dot menu (**...**), then select **Create file**. +1. Hover over the `models` directory and click the three-dot menu (**...**), then select **Create file**. 2. Name the file `staging/jaffle_shop/src_jaffle_shop.yml` , then click **Create**. 3. Copy the following text into the file and click **Save**. @@ -763,7 +755,7 @@ semantic_models: There are different types of metrics you can configure: - [Conversion metrics](/docs/build/conversion) — Track when a base event and a subsequent conversion event occur for an entity within a set time period. -- [Cumulative metrics](/docs/build/metrics-overview#cumulative-metrics) — Aggregate a measure over a given window. If no window is specified, the window will accumulate the measure over all of the recorded time period. Note, that you must create the time spine model before you add cumulative metrics. +- [Cumulative metrics](/docs/build/metrics-overview#cumulative-metrics) — Aggregate a measure over a given window. If no window is specified, the window will accumulate the measure over all of the recorded time period. Note that you must create the time spine model before you add cumulative metrics. - [Derived metrics](/docs/build/metrics-overview#derived-metrics) — Allows you to do calculations on top of metrics. - [Simple metrics](/docs/build/metrics-overview#simple-metrics) — Directly reference a single measure without any additional measures involved. - [Ratio metrics](/docs/build/metrics-overview#ratio-metrics) — Involve a numerator metric and a denominator metric. A constraint string can be applied to both the numerator and denominator or separately to the numerator or denominator. @@ -950,9 +942,9 @@ https://github.com/dbt-labs/docs.getdbt.com/blob/current/website/snippets/_sl-ru What’s happening internally? -- Merging the code into your main branch allows dbt Cloud to pull those changes and builds the definition in the manifest produced by the run.
+- Merging the code into your main branch allows dbt Cloud to pull those changes and build the definition in the manifest produced by the run.
- Re-running the job in the deployment environment helps materialize the models, which the metrics depend on, in the data platform. It also makes sure that the manifest is up to date.
-- The Semantic Layer APIs pulls in the most recent manifest and allows your integration information to extract metadata from it. +- The Semantic Layer APIs pull in the most recent manifest and enables your integration to extract metadata from it. @@ -965,7 +957,18 @@ https://github.com/dbt-labs/docs.getdbt.com/blob/current/website/snippets/_new-s -## Connect and query with Google Sheets +## Query the Semantic Layer + +This page will guide you on how to connect and use the following integrations to query your metrics: + +- [Connect and query with Google Sheets](#connect-and-query-with-google-sheets) +- [Connect and query with Hex](#connect-and-query-with-hex) + +The dbt Semantic Layer enables you to connect and query your metric with various available tools like Google Sheets, Hex, Tableau, and more. + +Query metrics using other tools such as [first-class integrations](/docs/cloud-integrations/avail-sl-integrations), [Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview), and [exports](/docs/use-dbt-semantic-layer/exports) to expose tables of metrics and dimensions in your data platform and create a custom integration with tools like PowerBI. + + ### Connect and query with Google Sheets ### Indirect selection examples - - To visualize these methods, suppose you have `model_a`, `model_b`, and `model_c` and associated data tests. The following illustrates which tests will be run when you execute `dbt build` with the various indirect selection modes: @@ -115,8 +109,6 @@ dbt build --select "orders" --indirect-selection=empty - - ### Test selection syntax examples diff --git a/website/docs/reference/node-selection/yaml-selectors.md b/website/docs/reference/node-selection/yaml-selectors.md index d911eb44baa..ff6628919b7 100644 --- a/website/docs/reference/node-selection/yaml-selectors.md +++ b/website/docs/reference/node-selection/yaml-selectors.md @@ -57,8 +57,6 @@ This is the most thorough syntax, which can include the operator-equivalent keyw Review [methods](/reference/node-selection/methods) for the available list. - - ```yml definition: method: tag @@ -77,9 +75,6 @@ definition: indirect_selection: eager | cautious | buildable | empty # include all tests selected indirectly? eager by default ``` - - - The `*` operator to select all nodes can be written as: ```yml definition: @@ -113,16 +108,11 @@ Note: The `exclude` argument in YAML selectors is subtly different from the `--exclude` CLI argument. Here, `exclude` _always_ returns a [set difference](https://en.wikipedia.org/wiki/Complement_(set_theory)), and it is always applied _last_ within its scope. - - When more than one "yeslist" (`--select`) is passed, they are treated as a [union](/reference/node-selection/set-operators#unions) rather than an [intersection](/reference/node-selection/set-operators#intersections). Same thing when there is more than one "nolist" (`--exclude`). - #### Indirect selection - - As a general rule, dbt will indirectly select _all_ tests if they touch _any_ resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious", "buildable", or "empty" by setting `indirect_selection` for a specific criterion: ```yml @@ -145,8 +135,6 @@ As a general rule, dbt will indirectly select _all_ tests if they touch _any_ re If provided, a YAML selector's `indirect_selection` value will take precedence over the CLI flag `--indirect-selection`. Because `indirect_selection` is defined separately for _each_ selection criterion, it's possible to mix eager/cautious/buildable/empty modes within the same definition, to achieve the exact behavior that you need. Remember that you can always test out your critiera with `dbt ls --selector`. - - See [test selection examples](/reference/node-selection/test-selection-examples) for more details about indirect selection. ## Example diff --git a/website/docs/reference/project-configs/config-version.md b/website/docs/reference/project-configs/config-version.md index c785b2d4c22..78b9be49a20 100644 --- a/website/docs/reference/project-configs/config-version.md +++ b/website/docs/reference/project-configs/config-version.md @@ -3,11 +3,7 @@ datatype: integer description: "Read this guide to understand the config-version configuration in dbt." --- - - -Starting in dbt v1.5, the `config-version:` tag is optional. - - +The `config-version:` tag is optional. @@ -18,9 +14,9 @@ config-version: 2 ## Definition -Specify your `dbt_project.yml` as using the v2 structure. - This configuration is optional. +Specify your `dbt_project.yml` as using the v2 structure. ## Default + Without this configuration, dbt will assume your `dbt_project.yml` uses the version 2 syntax. Version 1 was deprecated in dbt v0.19.0. diff --git a/website/docs/reference/project-configs/require-dbt-version.md b/website/docs/reference/project-configs/require-dbt-version.md index 5fb08927f0f..42dc49c4546 100644 --- a/website/docs/reference/project-configs/require-dbt-version.md +++ b/website/docs/reference/project-configs/require-dbt-version.md @@ -22,7 +22,7 @@ When you set this configuration, dbt sends a helpful error message for any user If this configuration is not specified, no version check will occur. -:::info Keep on latest version +:::info Versionless diff --git a/website/docs/reference/resource-configs/alias.md b/website/docs/reference/resource-configs/alias.md index e1d3ae41f8b..6cb14371dfa 100644 --- a/website/docs/reference/resource-configs/alias.md +++ b/website/docs/reference/resource-configs/alias.md @@ -63,27 +63,23 @@ seeds: -Configure a seed's alias in your `dbt_project.yml` file or config block. +Configure a snapshots's alias in your `dbt_project.yml` file or config block. -For example, if you have a snapshot that represents `your_snapshot` and want to alias it as `updated_at_id`, you would alias like this: +For example, if you have a snapshot that is named `your_snapshot` and want to alias it as `the_best_snapshot`, you would alias like this: ```yml snapshots: - - name: your_snapshot - config: - target_database: analytics - target_schema: finance - unique_key: id - strategy: timestamp - updated_at: updated_at - alias: your_snapshot + your_project: + your_snapshot: + +alias: the_best_snapshot ``` -This would return the name `analytics.finance.your_snapshot` in the database. +This would build your snapshot to `analytics.finance.the_best_snapshot` in the database. + @@ -100,8 +96,8 @@ models: columns: - name: order_id tests: - - unique - alias: unique_order_id_test + - unique: + alias: unique_order_id_test ``` When using `--store-failures`, this would return the name `analytics.finance.orders_order_id_unique_order_id_test` in the database. diff --git a/website/docs/reference/resource-configs/bigquery-configs.md b/website/docs/reference/resource-configs/bigquery-configs.md index 5a95c8f9567..851a93c2bcc 100644 --- a/website/docs/reference/resource-configs/bigquery-configs.md +++ b/website/docs/reference/resource-configs/bigquery-configs.md @@ -336,7 +336,7 @@ dbt supports the specification of BigQuery labels for the tables and BigQuery key-value pair entries for labels larger than 63 characters are truncated.
+ BigQuery key-value pair entries for labels larger than 63 characters are truncated. **Configuring labels in a model file** diff --git a/website/docs/reference/resource-configs/clickhouse-configs.md b/website/docs/reference/resource-configs/clickhouse-configs.md index 12dc0f5da3f..32f0c81f664 100644 --- a/website/docs/reference/resource-configs/clickhouse-configs.md +++ b/website/docs/reference/resource-configs/clickhouse-configs.md @@ -171,6 +171,8 @@ models: dbt snapshots allow a record to be made of changes to a mutable model over time. This in turn allows point-in-time queries on models, where analysts can “look back in time” at the previous state of a model. This functionality is supported by the ClickHouse connector and is configured using the following syntax: + + ```jinja @@ -186,14 +188,28 @@ dbt snapshots allow a record to be made of changes to a mutable model over time. -#### Snapshot Configuration + + + + + + +```jinja +{{ + config( + schema = "", + unique_key = "", + strategy = "", + updated_at = "", + ) +}} +``` + + + + -| Option | Description | Required? | -|-----------------|---------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| -| `target_schema` | A ClickHouse's database name where the snapshot table will be created. | Required | -| `unique_key` | A tuple of column names that uniquely identify rows. | Required. If not provided altered rows will be added twice to the incremental table. | -| `strategy` | Defines how dbt knows if a row has changed. More about dbt startegies [here](/docs/build/snapshots#detecting-row-changes) | Required | -| `updated_at` | If using the timestamp strategy, the timestamp column to compare. | Only if using the timestamp strategy | +For more information on configuration, check out the [snapshot configs](/reference/snapshot-configs) reference page. ## Supported Table Engines diff --git a/website/docs/reference/resource-configs/database.md b/website/docs/reference/resource-configs/database.md index 19c9eca272d..338159b30dc 100644 --- a/website/docs/reference/resource-configs/database.md +++ b/website/docs/reference/resource-configs/database.md @@ -45,6 +45,38 @@ This would result in the generated relation being located in the `staging` datab + + + + +Available for versionless dbt Cloud or dbt Core v1.9+. Select v1.9 or newer from the version dropdown to view the configs. + + + + + +Specify a custom database for a snapshot in your `dbt_project.yml` or config file. + +For example, if you have a snapshot that you want to load into a database other than the target database, you can configure it like this: + + + +```yml +snapshots: + your_project: + your_snapshot: + +database: snapshots +``` + + +This results in the generated relation being located in the `snapshots` database so the full relation name would be `snapshots.finance.your_snapshot` instead of the default target database. + + + + + + + Configure a database in your `dbt_project.yml` file. diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index fc2b2c81828..68224802026 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -7,20 +7,7 @@ id: "databricks-configs" When materializing a model as `table`, you may include several optional configs that are specific to the dbt-databricks plugin, in addition to the standard [model configs](/reference/model-configs). - - -| Option | Description | Required? | Example | -|---------------------|------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|--------------------------| -| file_format | The file format to use when creating tables (`parquet`, `delta`, `hudi`, `csv`, `json`, `text`, `jdbc`, `orc`, `hive` or `libsvm`). | Optional | `delta` | -| location_root | The created table uses the specified directory to store its data. The table alias is appended to it. | Optional | `/mnt/root` | -| partition_by | Partition the created table by the specified columns. A directory is created for each partition. | Optional | `date_day` | -| clustered_by | Each partition in the created table will be split into a fixed number of buckets by the specified columns. | Optional | `country_code` | -| buckets | The number of buckets to create while clustering | Required if `clustered_by` is specified | `8` | -| tblproperties | [Tblproperties](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-tblproperties.html) to be set on the created table | Optional | `{'this.is.my.key': 12}` | - - - - + | Option | Description | Required? | Model Support | Example | @@ -621,7 +608,7 @@ snapshots: - + ## Materialized views and streaming tables Starting with version 1.6.0, the dbt-databricks adapter supports [materialized views](https://docs.databricks.com/en/sql/user/materialized-views.html) and [streaming tables](https://docs.databricks.com/en/sql/load-data-streaming-table.html), as alternatives to incremental tables that are powered by [Delta Live Tables](https://docs.databricks.com/en/delta-live-tables/index.html). diff --git a/website/docs/reference/resource-configs/docs.md b/website/docs/reference/resource-configs/docs.md index aa19ae51888..e6b43b76060 100644 --- a/website/docs/reference/resource-configs/docs.md +++ b/website/docs/reference/resource-configs/docs.md @@ -161,7 +161,7 @@ macros: ## Definition -The docs field can be used to provide documentation-specific configuration to models. It supports the doc attribute `show`, which controls whether or not models are shown in the auto-generated documentation website. It also supports `node_color` for models, seeds, snapshots, and analyses. Other node types are not supported. +The `docs` property can be used to provide documentation-specific configuration to models. It supports the attribute `show`, which controls whether or not nodes are shown in the auto-generated documentation website. It also supports `node_color` for models, seeds, snapshots, and analyses. Other node types are not supported. **Note:** Hidden models will still appear in the dbt DAG visualization but will be identified as "hidden.” diff --git a/website/docs/reference/resource-configs/group.md b/website/docs/reference/resource-configs/group.md index 7b80d70b6fc..717d7de89f5 100644 --- a/website/docs/reference/resource-configs/group.md +++ b/website/docs/reference/resource-configs/group.md @@ -18,8 +18,6 @@ id: "group" }> - - ```yml @@ -60,14 +58,10 @@ select ... - - - - ```yml @@ -88,15 +82,10 @@ seeds: - - - - - ```yml @@ -123,15 +112,10 @@ select ... - - - - - ```yml @@ -184,8 +168,6 @@ select ... - - @@ -207,8 +189,6 @@ analyses: - - ```yaml @@ -233,8 +213,6 @@ metrics: - - diff --git a/website/docs/reference/resource-configs/oracle-configs.md b/website/docs/reference/resource-configs/oracle-configs.md index a050ba85b49..794fb840edc 100644 --- a/website/docs/reference/resource-configs/oracle-configs.md +++ b/website/docs/reference/resource-configs/oracle-configs.md @@ -36,7 +36,7 @@ from {{ source('sh_database', 'customers') }} c #### Querying -`COLUMN STORE COMPRESS FOR QUERY` is useful in data ware house environments. Valid values are `HIGH` or `LOW`, with `HIGH` providing a higher compression ratio. The default is `HIGH` +`COLUMN STORE COMPRESS FOR QUERY` is useful in data warehouse environments. Valid values are `HIGH` or `LOW`, with `HIGH` providing a higher compression ratio. The default is `HIGH` ```sql {{config(materialized='table', table_compression_clause='COLUMN STORE COMPRESS FOR QUERY LOW')}} @@ -54,7 +54,7 @@ from {{ source('sh_database', 'customers') }} c #### Archival -`COLUMN STORE COMPRESS FOR ARCHIVE` supports higher compression ratio than `COLUMN STORE COMPRESS FOR QUERY` and is useful for archival. Valid values are `HIGH` or `LOW` with `HIGH` providing the highest compression ratio. The default is `LOW` +`COLUMN STORE COMPRESS FOR ARCHIVE` supports a higher compression ratio than `COLUMN STORE COMPRESS FOR QUERY` and is useful for archival. Valid values are `HIGH` or `LOW` with `HIGH` providing the highest compression ratio. The default is `LOW` ```sql {{config(materialized='table', table_compression_clause='COLUMN STORE COMPRESS FOR ARCHIVE LOW')}} @@ -70,5 +70,48 @@ SELECT c.cust_id, c.cust_first_name, c.cust_last_name from {{ source('sh_database', 'customers') }} c ``` +## Partitioning + +Table and Incremental materialization configuration supports adding a partitioning clause: + +```sql +{ + config( + materialized='incremental', + unique_key='group_id', + parallel=4, + partition_config={"clause": "PARTITION BY HASH(PROD_NAME) PARTITIONS 4"}, + table_compression_clause='COLUMN STORE COMPRESS FOR QUERY LOW') +}} +SELECT * +FROM {{ source('sh_database', 'sales') }} +``` + +## Session info in `v$session` + +Custom session information can be supplied under `session_info` in `profile.yml` + + +```yaml +dbt_test: + target: dev + outputs: + dev: + type: oracle + user: "{{ env_var('DBT_ORACLE_USER') }}" + pass: "{{ env_var('DBT_ORACLE_PASSWORD') }}" + database: "{{ env_var('DBT_ORACLE_DATABASE') }}" + tns_name: "{{ env_var('DBT_ORACLE_TNS_NAME') }}" + schema: "{{ env_var('DBT_ORACLE_SCHEMA') }}" + threads: 4 + session_info: + action: "dbt run" + client_identifier: "dbt-unique-client-uuid" + client_info: "dbt Python3.9 thin driver" + module: "dbt-oracle-1.8.x" +``` + +This helps to track dbt sessions in the Database view [V$SESSION](https://docs.oracle.com/en/database/oracle/oracle-database/19/refrn/V-SESSION.html) + diff --git a/website/docs/reference/resource-configs/postgres-configs.md b/website/docs/reference/resource-configs/postgres-configs.md index b7b88c525dd..07cfc938f1c 100644 --- a/website/docs/reference/resource-configs/postgres-configs.md +++ b/website/docs/reference/resource-configs/postgres-configs.md @@ -8,22 +8,10 @@ id: "postgres-configs" In dbt-postgres, the following incremental materialization strategies are supported: - - -- `append` (default when `unique_key` is not defined) -- `delete+insert` (default when `unique_key` is defined) - - - - - - `append` (default when `unique_key` is not defined) - `merge` - `delete+insert` (default when `unique_key` is defined) - - - ## Performance optimizations ### Unlogged @@ -104,8 +92,6 @@ models: - - ## Materialized views The Postgres adapter supports [materialized views](https://www.postgresql.org/docs/current/rules-materializedviews.html) @@ -200,7 +186,7 @@ This happens via a `DROP/CREATE` of the indexes, which can be thought of as an ` Learn more about these parameters in Postgres's [docs](https://www.postgresql.org/docs/current/sql-creatematerializedview.html). - + ### Limitations @@ -216,5 +202,3 @@ If the user changes the model's config to `materialized="materialized_view"`, th The solution is to execute `DROP TABLE my_model` on the data warehouse before trying the model again. - - diff --git a/website/docs/reference/resource-configs/redshift-configs.md b/website/docs/reference/resource-configs/redshift-configs.md index 78b288083fa..dcd87118d13 100644 --- a/website/docs/reference/resource-configs/redshift-configs.md +++ b/website/docs/reference/resource-configs/redshift-configs.md @@ -14,21 +14,10 @@ To-do: In dbt-redshift, the following incremental materialization strategies are supported: - - -- `append` (default when `unique_key` is not defined) -- `delete+insert` (default when `unique_key` is defined) - - - - - - `append` (default when `unique_key` is not defined) - `merge` - `delete+insert` (default when `unique_key` is defined) - - All of these strategies are inherited from dbt-postgres. ## Performance optimizations @@ -107,8 +96,6 @@ models: - - ## Materialized views The Redshift adapter supports [materialized views](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-overview.html) @@ -241,7 +228,7 @@ As with most data platforms, there are limitations associated with materialized Find more information about materialized view limitations in Redshift's [docs](https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-create-sql-command.html#mv_CREATE_MATERIALIZED_VIEW-limitations). - + #### Changing materialization from "materialized_view" to "table" or "view" @@ -256,8 +243,6 @@ The workaround is to execute `DROP MATERIALIZED VIEW my_mv CASCADE` on the data - - ## Unit test limitations diff --git a/website/docs/reference/resource-configs/schema.md b/website/docs/reference/resource-configs/schema.md index 3852ee4e639..5a1a61d3943 100644 --- a/website/docs/reference/resource-configs/schema.md +++ b/website/docs/reference/resource-configs/schema.md @@ -8,9 +8,9 @@ datatype: string -Specify a custom schema for a group of models in your `dbt_project.yml` file or a [config block](/reference/resource-configs/schema#models). - -For example, if you have a group of marketing-related models and you want to place them in a separate schema called `marketing`, you can configure it like this: +Specify a [custom schema](/docs/build/custom-schemas#understanding-custom-schemas) for a group of models in your `dbt_project.yml` file or a [config block](/reference/resource-configs/schema#models). + +For example, if you have a group of marketing-related models and want to place them in a separate schema called `marketing`, you can configure it like this: @@ -22,7 +22,8 @@ models: ``` -This would result in the generated relations for these models being located in the `marketing` schema, so the full relation names would be `analytics.marketing.model_name`. +This would result in the generated relations for these models being located in the `marketing` schema, so the full relation names would be `analytics.target_schema_marketing.model_name`. This is because the schema of the relation is `{{ target.schema }}_{{ schema }}`. The [definition](#definition) section explains this in more detail. + @@ -40,10 +41,40 @@ seeds: +schema: mappings ``` -This would result in the generated relation being located in the `mappings` schema, so the full relation name would be `analytics.mappings.product_mappings`. +This would result in the generated relation being located in the `mappings` schema, so the full relation name would be `analytics.target_schema_mappings.product_mappings`. + + + + +Available for versionless dbt Cloud or dbt Core v1.9+. Select v1.9 or newer from the version dropdown to view the configs. + + + + + +Specify a custom schema for a snapshot in your `dbt_project.yml` or config file. + +For example, if you have a snapshot that you want to load into a schema other than the target schema, you can configure it like this: + + + +```yml +snapshots: + your_project: + your_snapshot: + +schema: snapshots +``` + + +This results in the generated relation being located in the `snapshots` schema so the full relation name would be `analytics.snapshots.your_snapshot` instead of the default target schema. + + + + + Customize the schema for storing test results in your `dbt_project.yml` file. @@ -118,17 +149,28 @@ seeds: ### Tests -Customize the name of the schema in which tests [configured to store failures](/reference/resource-configs/store_failures) will save their results: +Customize the name of the schema in which tests [configured to store failures](/reference/resource-configs/store_failures) will save their results. +The resulting schema is `{{ profile.schema }}_{{ tests.schema }}`, with a default suffix of `dbt_test__audit`. +To use the same profile schema, set `+schema: null`. ```yml tests: +store_failures: true - +schema: the_island_of_misfit_tests + +schema: _sad_test_failures # Will write tables to my_database.my_schema__sad_test_failures ``` +Ensure you have the authorization to create or access schemas for your work. To ensure that the required schemas have the correct permissions, run a sql statement in your respective data platform environment. For example, run the following command if using Redshift (exact authorization query may differ from one data platform to another): + +```sql +create schema if not exists dev_username_dbt_test__audit authorization username; +``` +_Replace `dev_username` with your specific development schema name and `username` with the appropriate user who should have the permissions._ + +This command grants the appropriate permissions to create and access the `dbt_test__audit` schema, which is often used with the `store_failures` configuration. + ## Warehouse specific information * BigQuery: `dataset` and `schema` are interchangeable diff --git a/website/docs/reference/resource-configs/singlestore-configs.md b/website/docs/reference/resource-configs/singlestore-configs.md index 2317c05c0e2..c1a1c6869f1 100644 --- a/website/docs/reference/resource-configs/singlestore-configs.md +++ b/website/docs/reference/resource-configs/singlestore-configs.md @@ -107,8 +107,6 @@ select ... - - ## Model contracts Starting from 1.5, the `dbt-singlestore` adapter supports model contracts. @@ -217,5 +215,3 @@ It's important to note that certain data type mappings might show up differently Just keep these points in mind when setting up and using your `dbt-singlestore` adapter, and you'll avoid common pitfalls! - - diff --git a/website/docs/reference/resource-configs/snowflake-configs.md b/website/docs/reference/resource-configs/snowflake-configs.md index 9d70d28eae8..a59bc8dee00 100644 --- a/website/docs/reference/resource-configs/snowflake-configs.md +++ b/website/docs/reference/resource-configs/snowflake-configs.md @@ -338,8 +338,6 @@ In the configuration format for the model SQL file: - - ## Dynamic tables The Snowflake adapter supports [dynamic tables](https://docs.snowflake.com/en/user-guide/dynamic-tables-about). @@ -351,11 +349,28 @@ of [materialized views](/docs/build/materializations#Materialized-View). In particular, dynamic tables have access to the `on_configuration_change` setting. Dynamic tables are supported with the following configuration parameters: -| Parameter | Type | Required | Default | Change Monitoring Support | -|----------------------------------------------------------------------------------|------------|----------|---------|---------------------------| -| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `` | no | `apply` | n/a | -| [`target_lag`](#target-lag) | `` | yes | | alter | -| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `` | yes | | alter | + + +| Parameter | Type | Required | Default | Change Monitoring Support | +|--------------------|------------|----------|-------------|---------------------------| +| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `` | no | `apply` | n/a | +| [`target_lag`](#target-lag) | `` | yes | | alter | +| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `` | yes | | alter | + + + + +| Parameter | Type | Required | Default | Change Monitoring Support | +|--------------------|------------|----------|-------------|---------------------------| +| [`on_configuration_change`](/reference/resource-configs/on_configuration_change) | `` | no | `apply` | n/a | +| [`target_lag`](#target-lag) | `` | yes | | alter | +| [`snowflake_warehouse`](#configuring-virtual-warehouses) | `` | yes | | alter | +| [`refresh_mode`](#refresh-mode) | `` | no | `AUTO` | refresh | +| [`initialize`](#initialize) | `` | no | `ON_CREATE` | n/a | + + + + + + + + +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[materialized](/reference/resource-configs/materialized): dynamic_table + [+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail + [+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream | + [+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses): + +``` + + + + + + + + + + +```yaml +version: 2 + +models: + - name: [] + config: + [materialized](/reference/resource-configs/materialized): dynamic_table + [on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail + [target_lag](#target-lag): downstream | + [snowflake_warehouse](#configuring-virtual-warehouses): + +``` + + + + + + + + + + +```jinja + +{{ config( + [materialized](/reference/resource-configs/materialized)="dynamic_table", + [on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail", + [target_lag](#target-lag)="downstream" | " seconds | minutes | hours | days", + [snowflake_warehouse](#configuring-virtual-warehouses)="", + +) }} + +``` + + + + + + + + + + + + @@ -379,6 +470,9 @@ models: [+](/reference/resource-configs/plus-prefix)[on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail [+](/reference/resource-configs/plus-prefix)[target_lag](#target-lag): downstream | [+](/reference/resource-configs/plus-prefix)[snowflake_warehouse](#configuring-virtual-warehouses): + [+](/reference/resource-configs/plus-prefix)[refresh_mode](#refresh-mode): AUTO | FULL | INCREMENTAL + [+](/reference/resource-configs/plus-prefix)[initialize](#initialize): ON_CREATE | ON_SCHEDULE + ``` @@ -400,6 +494,9 @@ models: [on_configuration_change](/reference/resource-configs/on_configuration_change): apply | continue | fail [target_lag](#target-lag): downstream | [snowflake_warehouse](#configuring-virtual-warehouses): + [refresh_mode](#refresh-mode): AUTO | FULL | INCREMENTAL + [initialize](#initialize): ON_CREATE | ON_SCHEDULE + ``` @@ -412,12 +509,17 @@ models: ```jinja + {{ config( [materialized](/reference/resource-configs/materialized)="dynamic_table", [on_configuration_change](/reference/resource-configs/on_configuration_change)="apply" | "continue" | "fail", [target_lag](#target-lag)="downstream" | " seconds | minutes | hours | days", [snowflake_warehouse](#configuring-virtual-warehouses)="", + [refresh_mode](#refresh-mode)="AUTO" | "FULL" | "INCREMENTAL", + [initialize](#initialize)="ON_CREATE" | "ON_SCHEDULE", + ) }} + ``` @@ -426,6 +528,8 @@ models: + + Learn more about these parameters in Snowflake's [docs](https://docs.snowflake.com/en/sql-reference/sql/create-dynamic-table): ### Target lag @@ -436,6 +540,27 @@ Snowflake allows two configuration scenarios for scheduling automatic refreshes: Learn more about `target_lag` in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh#understanding-target-lag). + + +### Refresh mode + +Snowflake allows three options for refresh mode: +- **AUTO** — Enforces an incremental refresh of the dynamic table by default. If the `CREATE DYNAMIC TABLE` statement does not support the incremental refresh mode, the dynamic table is automatically created with the full refresh mode. +- **FULL** — Enforces a full refresh of the dynamic table, even if the dynamic table can be incrementally refreshed. +- **INCREMENTAL** — Enforces an incremental refresh of the dynamic table. If the query that underlies the dynamic table can’t perform an incremental refresh, dynamic table creation fails and displays an error message. + +Learn more about `refresh_mode` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh). + +### Initialize + +Snowflake allows two options for initialize: +- **ON_CREATE** — Refreshes the dynamic table synchronously at creation. If this refresh fails, dynamic table creation fails and displays an error message. +- **ON_SCHEDULE** — Refreshes the dynamic table at the next scheduled refresh. + +Learn more about `initialize` in [Snowflake's docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-refresh). + + + ### Limitations As with materialized views on most data platforms, there are limitations associated with dynamic tables. Some worth noting include: @@ -447,7 +572,11 @@ As with materialized views on most data platforms, there are limitations associa Find more information about dynamic table limitations in Snowflake's [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-tasks-create#dynamic-table-limitations-and-supported-functions). - +For dbt limitations, these dbt features are not supported: +- [Model contracts](/docs/collaborate/govern/model-contracts) +- [Copy grants configuration](/reference/resource-configs/snowflake-configs#copying-grants) + + #### Changing materialization to and from "dynamic_table" @@ -474,9 +603,6 @@ The workaround is to execute `DROP TABLE my_model` on the data warehouse before - - - ## Source freshness known limitation Snowflake calculates source freshness using information from the `LAST_ALTERED` column, meaning it relies on a field updated whenever any object undergoes modification, not only data updates. No action must be taken, but analytics teams should note this caveat. diff --git a/website/docs/reference/resource-configs/store_failures.md b/website/docs/reference/resource-configs/store_failures.md index 042a587ab64..26b5b372701 100644 --- a/website/docs/reference/resource-configs/store_failures.md +++ b/website/docs/reference/resource-configs/store_failures.md @@ -10,15 +10,13 @@ Optionally set a test to always or never store its failures in the database. - If specified as `true` or `false`, the `store_failures` config will take precedence over the presence or absence of the `--store-failures` flag. - If the `store_failures` config is `none` or omitted, the resource will use the value of the `--store-failures` flag. -- When true, `store_failures` saves all the record(s) that failed the test only if [limit](/reference/resource-configs/limit) is not set or if there are fewer records than the limit. `store_failures` are saved in a new table with the name of the test. +- When true, `store_failures` saves all records (up to [limit](/reference/resource-configs/limit)) that failed the test. Failures are saved in a new table with the name of the test. By default, `store_failures` uses the schema `{{ profile.schema }}_dbt_test__audit`, but you can [configure](/reference/resource-configs/schema#tests) the schema suffix to a different value. - A test's results will always **replace** previous failures for the same test, even if that test results in no failures. -- By default, `store_failures` uses a schema named `dbt_test__audit`, but, you can [configure](/reference/resource-configs/schema#tests) the schema to a different value. +- By default, `store_failures` uses a schema named `dbt_test__audit`, but, you can [configure](/reference/resource-configs/schema#tests) the schema to a different value. Ensure you have the authorization to create or access schemas for your work. For more details, refer to the [FAQ](#faqs). This logic is encoded in the [`should_store_failures()`](https://github.com/dbt-labs/dbt-adapters/blob/60005a0a2bd33b61cb65a591bc1604b1b3fd25d5/dbt/include/global_project/macros/materializations/configs.sql#L15) macro. - - + +## FAQs + + + +If you're receiving a `Adapter name adapter: Adapter_name error: permission denied for schema dev_username_dbt_test__audit`, this is most likely due to your user not having permission to create new schemas, despite having owner access to your own development schema. + +To resolve this, you need proper authorization to create or access custom schemas. Run the following SQL command in your respective data platform environment. Note that the exact authorization query may differ from one data platform to another: + +```sql +create schema if not exists dev_username_dbt_test__audit authorization username; +``` +_Replace `dev_username` with your specific development schema name and `username` with the appropriate user who should have the permissions._ + +This command grants the appropriate permissions to create and access the `dbt_test__audit` schema, which is often used with the `store_failures` configuration. + + diff --git a/website/docs/reference/resource-configs/target_database.md b/website/docs/reference/resource-configs/target_database.md index 07837824f33..3c07b442107 100644 --- a/website/docs/reference/resource-configs/target_database.md +++ b/website/docs/reference/resource-configs/target_database.md @@ -4,6 +4,12 @@ description: "Target_database - Read this in-depth guide to learn about configur datatype: string --- +:::note + +For [versionless](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#versionless) dbt Cloud accounts and dbt Core v1.9+, this functionality is no longer utilized. Use the [database](/reference/resource-configs/database) config as an alternative to define a custom database while still respecting the `generate_database_name` macro. + +::: + ```yml diff --git a/website/docs/reference/resource-configs/target_schema.md b/website/docs/reference/resource-configs/target_schema.md index 9d459b32bad..893686a7513 100644 --- a/website/docs/reference/resource-configs/target_schema.md +++ b/website/docs/reference/resource-configs/target_schema.md @@ -4,6 +4,12 @@ description: "Target_schema - Read this in-depth guide to learn about configurat datatype: string --- +:::note + +For [versionless](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8#versionless) dbt Cloud accounts and dbt Core v1.9+, this functionality is no longer required. Use the [schema](/reference/resource-configs/schema) config as an alternative to define a custom schema while still respecting the `generate_schema_name` macro. + +::: + ```yml @@ -34,9 +40,6 @@ On **BigQuery**, this is analogous to a `dataset`. ## Default This is a **required** parameter, no default is provided. -## FAQs - - ## Examples ### Build all snapshots in a schema named `snapshots` diff --git a/website/docs/reference/resource-properties/unit-tests.md b/website/docs/reference/resource-properties/unit-tests.md index 3c6e6eef0a6..08081c4c24a 100644 --- a/website/docs/reference/resource-properties/unit-tests.md +++ b/website/docs/reference/resource-properties/unit-tests.md @@ -7,7 +7,7 @@ datatype: test :::note -This functionality is only supported in dbt Core v1.8+ or dbt Cloud accounts that have gone versionless by opting to ["Keep on latest version"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). +This functionality is only supported in dbt Core v1.8+ or dbt Cloud accounts that have gone ["Versionless"](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless). ::: diff --git a/website/docs/reference/snapshot-configs.md b/website/docs/reference/snapshot-configs.md index d487219a69f..5afe429cfb4 100644 --- a/website/docs/reference/snapshot-configs.md +++ b/website/docs/reference/snapshot-configs.md @@ -35,6 +35,8 @@ Parts of a snapshot: }> + + ```yaml @@ -51,37 +53,40 @@ snapshots: - + - - -**Note:** Required snapshot properties _will not_ work when defined in `config` YAML blocks. We recommend that you define these in `dbt_project.yml` or a `config()` block within the snapshot `.sql` file. + + - + + + + + + + +**Note:** Required snapshot properties _will not_ work when defined in `config` YAML blocks. We recommend that you define these in `dbt_project.yml` or a `config()` block within the snapshot `.sql` file. + ```jinja @@ -96,6 +101,25 @@ snapshots: ``` + + + + +```jinja + +{{ config( + [schema](/reference/resource-configs/schema)="", + [database](/reference/resource-configs/database)="", + [alias](/reference/resource-configs/alias)="", + [unique_key](/reference/resource-configs/unique_key)="", + [strategy](/reference/resource-configs/strategy)="timestamp" | "check", + [updated_at](/reference/resource-configs/updated_at)="", + [check_cols](/reference/resource-configs/check_cols)=[""] | "all" +) }} + +``` + + @@ -189,7 +213,7 @@ Snapshots can be configured in one of three ways: Snapshot configurations are applied hierarchically in the order above. ### Examples -#### Apply the `target_schema` configuration to all snapshots +#### Apply configurations to all snapshots To apply a configuration to all snapshots, including those in any installed [packages](/docs/build/packages), nest the configuration directly under the `snapshots` key: @@ -197,14 +221,14 @@ To apply a configuration to all snapshots, including those in any installed [pac ```yml snapshots: - +target_schema: snapshots + +unique_key: id ``` -#### Apply the `target_schema` configuration to all snapshots in your project -To apply a configuration to all snapshots in your project only (i.e. _excluding_ any snapshots in installed packages), provide your project name as part of the resource path. +#### Apply configurations to all snapshots in your project +To apply a configuration to all snapshots in your project only (for example, _excluding_ any snapshots in installed packages), provide your project name as part of the resource path. For a project named `jaffle_shop`: @@ -214,7 +238,7 @@ For a project named `jaffle_shop`: snapshots: jaffle_shop: - +target_schema: snapshot_data + +unique_key: id ``` diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js index 1ee1a308546..9eb040e54fb 100644 --- a/website/docusaurus.config.js +++ b/website/docusaurus.config.js @@ -81,7 +81,7 @@ var siteSettings = { "https://www.getdbt.com/resources/webinars/dbt-cloud-demos-with-experts/?utm_medium=internal&utm_source=docs&utm_campaign=q2-2025_biweekly-demos_aw&utm_content=biweekly-demos____&utm_term=all_all__", // Set community spotlight member on homepage // This is the ID for a specific file under docs/community/spotlight - communitySpotlightMember: "tyler-rouze", + communitySpotlightMember: "meagan-palmer", prism: { theme: (() => { var theme = require("prism-react-renderer/themes/nightOwl"); diff --git a/website/sidebars.js b/website/sidebars.js index 07f552088d0..7bb8a43d85b 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -59,6 +59,7 @@ const sidebarSettings = { }, items: [ "docs/cloud/connect-data-platform/about-connections", + "docs/cloud/connect-data-platform/connect-amazon-athena", "docs/cloud/connect-data-platform/connect-azure-synapse-analytics", "docs/cloud/connect-data-platform/connect-microsoft-fabric", "docs/cloud/connect-data-platform/connect-starburst-trino", @@ -79,6 +80,7 @@ const sidebarSettings = { items: [ "docs/cloud/manage-access/about-user-access", "docs/cloud/manage-access/invite-users", + "docs/cloud/manage-access/mfa", { type: "category", label: "User permissions and licenses", @@ -92,7 +94,18 @@ const sidebarSettings = { "docs/cloud/manage-access/enterprise-permissions", ], }, - + { + type: "category", + label: "Environment permissions", + link: { + type: "doc", + id: "docs/cloud/manage-access/environment-permissions", + }, + items: [ + "docs/cloud/manage-access/environment-permissions", + "docs/cloud/manage-access/environment-permissions-setup", + ], + }, { type: "category", label: "Single sign-on and Oauth", @@ -451,6 +464,7 @@ const sidebarSettings = { "docs/deploy/job-scheduler", "docs/deploy/deploy-environments", "docs/deploy/continuous-integration", + "docs/deploy/continuous-deployment", { type: "category", label: "Jobs", @@ -669,6 +683,7 @@ const sidebarSettings = { "docs/dbt-cloud-apis/sl-api-overview", "docs/dbt-cloud-apis/sl-jdbc", "docs/dbt-cloud-apis/sl-graphql", + "docs/dbt-cloud-apis/sl-python", "docs/dbt-cloud-apis/sl-manifest", ], }, diff --git a/website/snippets/_adapters-trusted.md b/website/snippets/_adapters-trusted.md index 178f1e3224b..2ac5268fc28 100644 --- a/website/snippets/_adapters-trusted.md +++ b/website/snippets/_adapters-trusted.md @@ -14,7 +14,7 @@ diff --git a/website/snippets/_cloud-environments-info.md b/website/snippets/_cloud-environments-info.md index 5013f9763ff..166165be855 100644 --- a/website/snippets/_cloud-environments-info.md +++ b/website/snippets/_cloud-environments-info.md @@ -35,7 +35,7 @@ Both development and deployment environments have a section called **General Set - dbt Cloud allows users to select any dbt release. At this time, **environments must use a dbt version greater than or equal to v1.0.0;** [lower versions are no longer supported](/docs/dbt-versions/upgrade-dbt-version-in-cloud). - If you select a current version with `(latest)` in the name, your environment will automatically install the latest stable version of the minor version selected. -- Go versionless by opting to **Keep on latest version**, which removes the need for manually upgrading environment, while ensuring you are always up to date with the latest fixes and features. +- Go **Versionless**, which removes the need for manually upgrading environment, while ensuring you are always up to date with the latest fixes and features. ::: ### Custom branch behavior diff --git a/website/snippets/_config-dbt-version-check.md b/website/snippets/_config-dbt-version-check.md index b5283aae864..d4e495bd379 100644 --- a/website/snippets/_config-dbt-version-check.md +++ b/website/snippets/_config-dbt-version-check.md @@ -1,5 +1,5 @@ -Starting in 2024, when you select **Keep on latest version** in dbt Cloud, dbt will ignore the `require-dbt-version` config. Refer to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version) for more details about going versionless. +Starting in 2024, when you select **Versionless** in dbt Cloud, dbt will ignore the `require-dbt-version` config. Refer to [Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) for more details. dbt Labs is committed to zero breaking changes for code in dbt projects, with ongoing releases to dbt Cloud and new versions of dbt Core. We also recommend these best practices: diff --git a/website/snippets/_enterprise-permissions-table.md b/website/snippets/_enterprise-permissions-table.md index a21a1869498..cef68e894f5 100644 --- a/website/snippets/_enterprise-permissions-table.md +++ b/website/snippets/_enterprise-permissions-table.md @@ -7,55 +7,55 @@ Key: Permissions: * Account-level permissions — Permissions related to the management of the dbt Cloud account. For example, billing and account settings. -* Project-level permissions — Permissions related to the projects in dbt Cloud. For example, repos and access to the IDE or dbt Cloud CLI. +* Project-level permissions — Permissions related to the projects in dbt Cloud. For example, repos and access to the dbt Cloud IDE or dbt Cloud CLI. ### Account roles Account roles enable you to manage the dbt Cloud account and manage the account settings (for example, generating service tokens, inviting users, and configuring SSO). They also provide project-level permissions. The **Account Admin** role is the highest level of access you can assign. #### Account permissions for account roles - -| Account-level permission| Account Admin | Billing admin | Project creator | Security admin | Viewer | -|:-------------------------|:-------------:|:-------------:|:---------------:|:--------------:|:------:| -| Account settings | W | | R | R | R | -| Audit logs | R | | | R | R | -| Auth provider | W | | | W | R | -| Billing | W | W | | | R | -| Groups | W | | R | W | R | -| Invitations | W | | W | W | R | -| IP restrictions | W | | | W | R | -| Licenses | W | | W | W | R | -| Members | W | | W | W | R | -| Project (create) | W | | W | | | -| Public models | R | R | R | R | R | -| Service tokens | W | | | R | R | -| Webhooks | W | | | | | +| Account-level permission| Account Admin | Billing admin | Manage

marketplace

apps | Project creator | Security admin | Viewer | +|:-------------------------|:-------------:|:------------:|:-------------------------:|:---------------:|:--------------:|:------:| +| Account settings | W | | | R | R | R | +| Audit logs | R | | | | R | R | +| Auth provider | W | | | | W | R | +| Billing | W | W | | | | R | +| Groups | W | | | R | W | R | +| Invitations | W | | | W | W | R | +| IP restrictions | W | | | | W | R | +| Licenses | W | | | W | W | R | +| Marketplace app | | | W | | | | +| Members | W | | | W | W | R | +| Project (create) | W | | | W | | | +| Public models | R | R | | R | R | R | +| Service tokens | W | | | | R | R | +| Webhooks | W | | | | | | #### Project permissions for account roles - + |Project-level permission | Account Admin | Billing admin | Project creator | Security admin | Viewer | |:-------------------------|:-------------:|:-------------:|:---------------:|:--------------:|:------:| -| Connections | W | | W | | R | -| Credentials | W | | W | | R | +| Data platform connections | W | | W | | R | +| Environment credentials (deployment) | W | | W | | R | | Custom env. variables | W | | W | | R | -| dbt adapters | W | | W | | R | +| Data platform configurations | W | | W | | R | | Develop (IDE or dbt Cloud CLI) | W | | W | | | | Environments | W | | W | | R | | Jobs | W | | W | | R | -| Metadata | R | | R | | R | +| Metadata GraphQL API access | R | | R | | R | | Permissions | W | | W | W | R | | Profile | W | | W | | R | | Projects | W | | W | R | R | | Repositories | W | | W | | R | | Runs | W | | W | | R | -| Semantic Layer Config | W | | W | | R | +| Semantic Layer config | W | | W | | R | ### Project role permissions - + The project roles enable you to work within the projects in various capacities. They primarily provide access to project-level permissions such as repos and the IDE or dbt Cloud CLI, but may also provide some account-level permissions. #### Account permissions for project roles - + | Account-level permission | Admin | Analyst | Database admin | Developer | Git Admin | Job admin | Job runner | Job viewer | Metadata

(Discovery API only) | Semantic Layer | Stakeholder | Team admin | Webhook | |--------------------------|:-----:|:-------:|:--------------:|:---------:|:---------:|:---------:|:-----------:|:-----------:|:--------:|:--------------:|:-----------:|:----------:|:-------:| | Account settings | R | | R | | R | | | | | | | R | | @@ -71,20 +71,20 @@ The project roles enable you to work within the projects in various capacities. | Webhooks | W | | | W | | | | | | | | | W | #### Project permissions for project roles - + |Project-level permission | Admin | Analyst | Database admin | Developer | Git Admin | Job admin | Job runner | Job viewer | Metadata

(Discovery API only) | Semantic Layer | Stakeholder | Team admin | Webhook | |--------------------------|:-----:|:-------:|:--------------:|:---------:|:---------:|:---------:|:-----------:|:-----------:|:--------:|:--------------:|:-----------:|:----------:|:-------:| -| Connections | W | R | W | R | R | R | | | | | R | R | | -| Credentials | W | W | W | W | R | W | | | | | R | R | | +| Data platform connections | W | R | W | R | R | R | | | | | R | R | | +| Environment credentials (deployment) | W | W | W | W | R | W | | | | | R | R | | | Custom env. variables | W | W | W | W | W | W | | R | | | R | W | | -| dbt adapters | W | W | W | W | R | W | | | | | R | R | | +| Data platform configurations | W | W | W | W | R | W | | | | | R | R | | | Develop
(IDE or dbt Cloud CLI) | W | W | | W | | | | | | | | | | | Environments | W | R | R | R | R | W | | R | | | R | R | | | Jobs | W | R | R | W | R | W | R | R | | | R | R | | -| Metadata | R | R | R | R | R | R | | R | R | | R | R | | +| Metadata GraphQL API access | R | R | R | R | R | R | | R | R | | R | R | | | Permissions (Groups & Licenses) | W | | R | R | R | | | | | | | R | | | Profile (Credentials) | W | R | | R | R | R | | | | | R | | | | Projects | W | W | W | W | W | R | | R | | | R | W | | | Repositories | W | | R | R | W | | | | | | R | R | | | Runs | W | R | R | W | R | W | W | R | | | R | R | | -| Semantic Layer Config | W | R | W | R | R | R | | | | W | R | R | | +| Semantic Layer config | W | R | W | R | R | R | | | | W | R | R | | diff --git a/website/snippets/_mesh-cycle-detection.md b/website/snippets/_mesh-cycle-detection.md new file mode 100644 index 00000000000..2a48d0a15bd --- /dev/null +++ b/website/snippets/_mesh-cycle-detection.md @@ -0,0 +1,14 @@ +Currently, the default behavior for "project" dependencies enforces that these relationships only go in one direction, meaning that the `jaffle_finance` project could not add a new model that depends, on any public models produced by the `jaffle_marketing` project. dbt will check for cycles across projects and raise errors if any are detected. + +However, many teams may want to be able to share data assets back and forth between teams. _We've added support for enabling bidirectional dependencies across projects, currently in beta_. + +To enable this in your account, set the environment variable `DBT_CLOUD_PROJECT_CYCLES_ALLOWED` to `TRUE` in all your dbt Cloud environments. This allows you to create bidirectional dependencies between projects, so long as the new dependency does not introduce any node-level cycles. + +When setting up projects that depend on each other, it's important to do so in a stepwise fashion. Each project must run and produce public models before the original producer project can take a dependency on the original consumer project. For example, the order of operations would be as follows for a simple two-project setup: + +1. The `project_a` project runs in a deployment environment and produces public models. +2. The `project_b` project adds `project_a` as a dependency. +3. The `project_b` project runs in a deployment environment and produces public models. +4. The `project_a` project adds `project_b` as a dependency. + +If you enable this feature and experience any issues, please reach out to [dbt Cloud support](mailto:support@getdbt.com). diff --git a/website/snippets/_new-sl-setup.md b/website/snippets/_new-sl-setup.md index bd1410d89c6..383a1215f5b 100644 --- a/website/snippets/_new-sl-setup.md +++ b/website/snippets/_new-sl-setup.md @@ -1,31 +1,96 @@ -You can set up the dbt Semantic Layer in dbt Cloud at the environment and project level. Before you begin: +You must be part of the Owner group and have the correct [license](/docs/cloud/manage-access/seats-and-users) and [permissions](/docs/cloud/manage-access/self-service-permissions) to set up the Semantic Layer at the environment and project level. +- Enterprise plan: + - Developer license with Account Admin permissions, or + - Owner with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. +- Team plan: Owner with a Developer license. +- Free trial: You are on a free trial of the Team plan as an Owner, which means you have access to the dbt Semantic Layer. -- You must be part of the Owner group, and have the correct [license](/docs/cloud/manage-access/seats-and-users) and [permissions](/docs/cloud/manage-access/self-service-permissions) to configure the Semantic Layer: - * Enterprise plan — Developer license with Account Admin permissions. Or Owner with a Developer license, assigned Project Creator, Database Admin, or Admin permissions. - * Team plan — Owner with a Developer license. - * If you are using a free trial dbt Cloud account, you are on a trial of the Team plan as an Owner, so you're good to go. -- You must have a successful run in your new environment. +### 1. Select environment -Now that we've created and successfully run a job in your environment, you're ready to configure the semantic layer. +Select the environment where you want to enable the Semantic Layer: -1. Navigate to **Account Settings** in the navigation menu. -2. Use the sidebar to select your project settings. Select the specific project you want to enable the Semantic Layer for. -3. In the **Project Details** page, navigate to the **Semantic Layer** section, and select **Configure Semantic Layer**. +1. Navigate to **Account settings** in the navigation menu. +2. On the **Settings** left sidebar, select the specific project you want to enable the Semantic Layer for. +3. In the **Project details** page, navigate to the **Semantic Layer** section. Select **Configure Semantic Layer**. - + -4. In the **Set Up Semantic Layer Configuration** page, enter the credentials you want the Semantic Layer to use specific to your data platform. +4. In the **Set Up Semantic Layer Configuration** page, select the deployment environment you want for the Semantic Layer and click **Save**. This provides administrators with the flexibility to choose the environment where the Semantic Layer will be enabled. - - Use credentials with minimal privileges. This is because the Semantic Layer requires read access to the schema(s) containing the dbt models used in your semantic models for downstream applications - - Note, [Environment variables](/docs/build/environment-variables) such as `{{env_var('DBT_WAREHOUSE')}`, doesn't supported the dbt Semantic Layer yet. You must use the actual credentials. +:::tip dbt Cloud Enterprise can skip to [Add more credentials](#4-add-more-credentials) +dbt Cloud Enterprise plans can add multiple credentials and have a different set up. Skip to [Add more credentials](#4-add-more-credentials) for more configuration details. +::: - +### 2. Add a credential and create service tokens -5. Select the deployment environment you want for the Semantic Layer and click **Save**. -6. After saving it, you'll be provided with the connection information that allows you to connect to downstream tools. If your tool supports JDBC, save the JDBC URL or individual components (like environment id and host). If it uses the GraphQL API, save the GraphQL API host information instead. +The dbt Semantic Layer uses [service tokens](/docs/dbt-cloud-apis/service-tokens) for authentication which are tied to an underlying data platform credential that you configure. The credential configured is used to execute queries that the Semantic Layer issues against your data platform. This credential controls the physical access to underlying data accessed by the Semantic Layer, and all access policies set in the data platform for this credential will be respected. + +dbt Cloud Enterprise plans can add multiple credentials and map those to service tokens. Refer to [Add more credentials](#4-add-more-credentials) for more information. + +1. In the **Set Up Semantic Layer Configuration** page, enter the credentials specific to your data platform that you want the Semantic Layer to use. + - Use credentials with minimal privileges. The Semantic Layer requires read access to the schema(s) containing the dbt models used in your semantic models for downstream applications + - Note, environment variables such as `{{env_var('DBT_WAREHOUSE')}`, aren't supported in the dbt Semantic Layer yet. You must use the actual credentials. + + +2. Create a **Service Token** after you add the credential. + * Enterprise plans: Name and generate a service token on the credential page directly. + * Team plans: You can return to the **Project Details** page and click the **Generate a Service Token** button. +3. Name the token and save it. Once the token is generated, you won't be able to view this token again so make sure to record it somewhere safe. + +:::info +Teams plans can create multiple service tokens that map to one underlying credential. Adding [multiple credentials](#4-add-more-credentials) for tailored access is available for Enterprise plans. + +Book a free live demo to discover the full potential of dbt Cloud Enterprise. +::: + +### 3. View connection detail +1. Go back to the **Project details** page for connection details to connect to downstream tools. +2. Copy and share the environment ID, service token, host, as well as the service token name to the relevant teams for BI connection set up. If your tool uses the GraphQL API, save the GraphQL API host information instead of the JDBC URL. + + For info on how to connect to other integrations, refer to [Available integrations](/docs/cloud-integrations/avail-sl-integrations). -7. Save and copy your environment ID, service token, and host, which you'll need to use in the downstream tools. For more info on how to integrate with partner integrations, refer to [Available integrations](/docs/cloud-integrations/avail-sl-integrations). +### 4. Add more credentials +dbt Cloud Enterprise plans can optionally add multiple credentials and map them to service tokens, offering more granular control and tailored access for different teams, which can then be shared to relevant teams for BI connection setup. These credentials control the physical access to underlying data accessed by the Semantic Layer. + +We recommend configuring credentials and service tokens to reflect your teams and their roles. For example, create tokens or credentials that align with your team's needs, such as providing access to finance-related schemas to the Finance team. + +Note that: +- Admins can link multiple service tokens to a single credential within a project, but each service token can only be linked to one credential per project. +- When you send a request through the APIs, the service token of the linked credential will follow access policies of the underlying view and tables used to build your semantic layer requests. +- [Environment variables](/docs/build/environment-variables), like `{{env_var('DBT_WAREHOUSE')}` aren't supported the dbt Semantic Layer yet. You must use the actual credentials instead. + +To add multiple credentials and map them to service tokens: + +1. After configuring your environment, on the **Credentials & service tokens** page click the **Add Semantic Layer credential** button to configure the credential for your data platform. +2. On the **Create New Semantic Layer Credential** page, you can create multiple credentials and map them to a service token. +3. In the **Add credentials** section, fill in the data platform's credential fields. We recommend using “read-only” credentials. + +4. In the **Map new service token** section, map a service token to the credential you configured in the previous step. dbt Cloud automatically selects the service token permission set you need (Semantic Layer Only and Metadata Only). + - To add another service token, click **Add service token** under the **Linked service tokens** section. + +5. Click **Save** to link the service token to the credential. Remember to copy and save the service token securely, as it won't be viewable again after generation. + + +6. To delete a credential, go back to the **Semantic Layer & Credential**s page. Select **Delete credential** to remove a credential and click **Save**. + + When you delete a credential, any service tokens mapped to that credential in the project will no longer work and will break for any end users. + +## Additional configuration + + The following are the additional flexible configurations for Semantic Layer credentials. + +### Unlink service tokens +- Unlink a service token from the credential by clicking **Unlink** under the **Linked service tokens** section. If you try to query the Semantic Layer with an unlinked credential, you'll experience an error in your BI tool because no valid token is mapped. + +### Manage from service token page +**View credential from service token** +- View your Semantic Layer credential directly by navigating to the **API tokens** and then **Service tokens** page. +- Select the service token to view the credential it's linked to. This is useful if you want to know which service tokens are mapped to credentials in your project. + +**Create a new service token** +- From the **Service tokens** page, create a new service token and map it to the credential(s) (assuming the semantic layer permission exists). This is useful if you want to create a new service token and directly map it to a credential in your project. +- Make sure to select the correct permission set for the service token (Semantic Layer Only and Metadata Only). -8. Return to the **Project Details** page and click the **[Generate a Service Token](/docs/dbt-cloud-apis/service-tokens)** button. Make sure it has Semantic Layer Only and Metadata Only permissions. Name the token and save it. Once the token is generated, you won't be able to view this token again so make sure to record it somewhere safe. + diff --git a/website/snippets/_self-service-permissions-table.md b/website/snippets/_self-service-permissions-table.md index 101034a44ca..12f6577ddbd 100644 --- a/website/snippets/_self-service-permissions-table.md +++ b/website/snippets/_self-service-permissions-table.md @@ -27,10 +27,8 @@ Permissions: | Billing | W | | | | Groups | W | R | R | | Invitations | W | W | R | -| IP restrictions | W | | | | Licenses | W | R | | | Members | W | R | R | -| Private Link | W | R | R | | Project (create) | W | W | | | Public models | R | R | R | | Service tokens | W | | | diff --git a/website/snippets/_sl-connect-and-query-api.md b/website/snippets/_sl-connect-and-query-api.md index 43790ba7940..3bc4e6e1c96 100644 --- a/website/snippets/_sl-connect-and-query-api.md +++ b/website/snippets/_sl-connect-and-query-api.md @@ -1,7 +1,4 @@ -This section will guide you on how to use the Google Sheets integration. Query metrics using other tools: -- [First-class integrations](/docs/cloud-integrations/avail-sl-integrations) with Tableau, Hex, and more. -- The [dbt Semantic Layer APIs](/docs/dbt-cloud-apis/sl-api-overview). -- [Exports](/docs/use-dbt-semantic-layer/exports) to expose tables of metrics and dimensions in your data platform and create a custom integration with tools like PowerBI. +The Google Sheets integration allows you to query your metrics using Google Sheets. This section will guide you on how to connect and use the Google Sheets integration. To query your metrics using Google Sheets: 1. Make sure you have a [Gmail](http://gmail.com/) account. diff --git a/website/snippets/_sl-deprecation-notice.md b/website/snippets/_sl-deprecation-notice.md deleted file mode 100644 index 2c42dd199c7..00000000000 --- a/website/snippets/_sl-deprecation-notice.md +++ /dev/null @@ -1,5 +0,0 @@ -:::info Deprecation of dbt Metrics and the legacy dbt Semantic Layer -dbt Labs has deprecated dbt Metrics and the legacy dbt Semantic Layer, both supported on dbt version 1.5 or lower. These changes went into effect on December 15th, 2023. - -To migrate and access [MetricFlow](/docs/build/build-metrics-intro) or the re-released dbt Semantic Layer, use the [dbt Semantic Layer migration guide](/guides/sl-migration) and [upgrade your version](/docs/dbt-versions/upgrade-dbt-version-in-cloud) in dbt Cloud. -::: diff --git a/website/snippets/_v2-sl-prerequisites.md b/website/snippets/_v2-sl-prerequisites.md index f07c796936a..f8108849f4f 100644 --- a/website/snippets/_v2-sl-prerequisites.md +++ b/website/snippets/_v2-sl-prerequisites.md @@ -1,10 +1,7 @@ -- Have a dbt Cloud Team or Enterprise account. Suitable for both Multi-tenant and Single-tenant deployment. - - Note: Single-tenant accounts should contact their account representative for necessary setup and enablement. -- Have both your production and development environments running [dbt version 1.6 or higher](/docs/dbt-versions/upgrade-dbt-version-in-cloud). +- Have a dbt Cloud Team or Enterprise account. Single-tenant accounts should contact their account representative setup. +- Ensure your production and development environments use [dbt version 1.6 or higher](/docs/dbt-versions/upgrade-dbt-version-in-cloud). - Use Snowflake, BigQuery, Databricks, or Redshift. - Create a successful run in the environment where you configure the Semantic Layer. - - **Note:** Semantic Layer currently supports the Deployment environment for querying. (_development querying experience coming soon_) -- Set up the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) in the integrated tool to import metric definitions. - - dbt Core or Developer accounts can define metrics but won't be able to dynamically query them.
-- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the latest dbt Semantic Layer. -- Note that the dbt Semantic Layer doesn't yet support SSH tunneling for [Postgres or Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections. It also doesn't support using [Single sign-on (SSO)](/docs/cloud/manage-access/sso-overview) for Semantic Layer [production credentials](/docs/dbt-cloud-apis/service-tokens#permissions-for-service-account-tokens), however, SSO is supported for development user accounts. + - **Note:** Semantic Layer supports querying in Deployment environments; development querying is coming soon. +- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts powering the dbt Semantic Layer. +- Note that the dbt Semantic Layer doesn't support SSH tunneling for [Postgres or Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections. It also doesn't support using [Single sign-on (SSO)](/docs/cloud/manage-access/sso-overview) for [production credentials](/docs/dbt-cloud-apis/service-tokens#permissions-for-service-account-tokens), though SSO is supported for development user accounts. diff --git a/website/snippets/cloud-feature-parity.md b/website/snippets/cloud-feature-parity.md index 6c6f224f215..b6c49aa0faa 100644 --- a/website/snippets/cloud-feature-parity.md +++ b/website/snippets/cloud-feature-parity.md @@ -1,17 +1,17 @@ The following table outlines which dbt Cloud features are supported on the different SaaS options available today. For more information about feature availability, please [contact us](https://www.getdbt.com/contact/). -| Feature | Multi-tenant | AWS single tenant | Azure single tenant | -|-------------------------------|--------------|-----------------------|----------------------| -| Audit logs | ✅ | ✅ | ✅ | -| Continuous integration jobs | ✅ | ✅ | ✅ | -| dbt Cloud CLI | ✅ | ✅ | ✅ | -| dbt Cloud IDE | ✅ | ✅ | ✅ | -| dbt Explorer | ✅ | ✅ | ✅ | -| dbt Mesh | ✅ | ✅ | ✅ | -| dbt Semantic Layer | ✅ | ✅ (Upon request) | ❌ | -| Discovery API | ✅ | ✅ | ✅ | -| IP restrictions | ✅ | ✅ | ✅ | -| Job scheduler | ✅ | ✅ | ✅ | -| PrivateLink egress | ✅ (AWS only)| ✅ | ✅ | -| PrivateLink ingress | ❌ | ✅ | ✅ | -| Webhooks (Outbound) | ✅ | ✅ | ❌ | +| Feature | AWS Multi-tenant | AWS single tenant |Azure multi-tenant ([Preview](/docs/dbt-versions/product-lifecycles#dbt-cloud)) | Azure single tenant | +|-------------------------------|------------------|-----------------------|---------------------|------------------- -| +| Audit logs | ✅ | ✅ | ✅ | ✅ | +| Continuous integration jobs | ✅ | ✅ | ✅ | ✅ | +| dbt Cloud CLI | ✅ | ✅ | ✅ | ✅ | +| dbt Cloud IDE | ✅ | ✅ | ✅ | ✅ | +| dbt Explorer | ✅ | ✅ | ✅ | ✅ | +| dbt Mesh | ✅ | ✅ | ✅ | ✅ | +| dbt Semantic Layer | ✅ | ✅ (Upon request) | ✅ | ❌ | +| Discovery API | ✅ | ✅ | ✅ | ✅ | +| IP restrictions | ✅ | ✅ | ✅ | ✅ | +| Job scheduler | ✅ | ✅ | ✅ | ✅ | +| PrivateLink egress | ✅ (AWS only) | ✅ | ✅ | ✅ | +| PrivateLink ingress | ❌ | ✅ | ❌ | ✅ | +| Webhooks (Outbound) | ✅ | ✅ | ✅ | ❌ | diff --git a/website/snippets/core-versions-table.md b/website/snippets/core-versions-table.md index 13a77f6dd30..b9f79f7d76a 100644 --- a/website/snippets/core-versions-table.md +++ b/website/snippets/core-versions-table.md @@ -4,7 +4,7 @@ |:-------------------------------------------------------------:|:---------------:|:-------------------------------------:| | [**v1.8**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.8) | May 9 2024 | Active — May 8, 2025 | | [**v1.7**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.7) | Nov 2, 2023 | Critical — Nov 1, 2024 | -| [**v1.6**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.6) | Jul 31, 2023 | Critical — Jul 30, 2024 | +| [**v1.6**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.6) | Jul 31, 2023 | End of Life* ⚠️ | | [**v1.5**](/docs/dbt-versions/core-upgrade/upgrading-to-v1.5) | Apr 27, 2023 | End of Life* ⚠️ | | [**v1.4**](/docs/dbt-versions/core-upgrade/older%20versions/upgrading-to-v1.4) | Jan 25, 2023 | End of Life* ⚠️ | | [**v1.3**](/docs/dbt-versions/core-upgrade/older%20versions/upgrading-to-v1.3) | Oct 12, 2022 | End of Life* ⚠️ | diff --git a/website/src/components/communitySpotlightList/index.js b/website/src/components/communitySpotlightList/index.js index c52fe7891c7..8355fd0958b 100644 --- a/website/src/components/communitySpotlightList/index.js +++ b/website/src/components/communitySpotlightList/index.js @@ -11,7 +11,7 @@ const communityDescription = "The dbt Community is where analytics engineering l // This date determines where the 'Previously on the Spotlight" text will show. // Any spotlight members with a 'dateCreated' field before this date // will be under the 'Previously..' header. -const currentSpotlightDate = new Date('2024-05-01') +const currentSpotlightDate = new Date('2024-07-26') function CommunitySpotlightList({ spotlightData }) { const { siteConfig } = useDocusaurusContext() diff --git a/website/static/img/blog/authors/nathaniel_burren.jpeg b/website/static/img/blog/authors/nathaniel_burren.jpeg new file mode 100644 index 00000000000..ad556f8605a Binary files /dev/null and b/website/static/img/blog/authors/nathaniel_burren.jpeg differ diff --git a/website/static/img/community/spotlight/Meagan-Palmer.png b/website/static/img/community/spotlight/Meagan-Palmer.png new file mode 100644 index 00000000000..b24bf8db484 Binary files /dev/null and b/website/static/img/community/spotlight/Meagan-Palmer.png differ diff --git a/website/static/img/community/spotlight/Mikko-Sulonen.png b/website/static/img/community/spotlight/Mikko-Sulonen.png new file mode 100644 index 00000000000..faba53b639b Binary files /dev/null and b/website/static/img/community/spotlight/Mikko-Sulonen.png differ diff --git a/website/static/img/community/spotlight/Radovan-Bacovic.png b/website/static/img/community/spotlight/Radovan-Bacovic.png new file mode 100644 index 00000000000..f4e98bede9d Binary files /dev/null and b/website/static/img/community/spotlight/Radovan-Bacovic.png differ diff --git a/website/static/img/docs/cloud-integrations/welcome_to_hex.png b/website/static/img/docs/cloud-integrations/welcome_to_hex.png new file mode 100644 index 00000000000..8d588484600 Binary files /dev/null and b/website/static/img/docs/cloud-integrations/welcome_to_hex.png differ diff --git a/website/static/img/docs/collaborate/dbt-explorer/example-performance-overview-page.png b/website/static/img/docs/collaborate/dbt-explorer/example-performance-overview-page.png index 4b6d841294c..f7e569c3252 100644 Binary files a/website/static/img/docs/collaborate/dbt-explorer/example-performance-overview-page.png and b/website/static/img/docs/collaborate/dbt-explorer/example-performance-overview-page.png differ diff --git a/website/static/img/docs/collaborate/dbt-explorer/example-prop-inherit.jpg b/website/static/img/docs/collaborate/dbt-explorer/example-prop-inherit.jpg new file mode 100644 index 00000000000..425725f1149 Binary files /dev/null and b/website/static/img/docs/collaborate/dbt-explorer/example-prop-inherit.jpg differ diff --git a/website/static/img/docs/collaborate/dbt-explorer/example-prop-inherit.png b/website/static/img/docs/collaborate/dbt-explorer/example-prop-inherit.png deleted file mode 100644 index 696ef5b9eb1..00000000000 Binary files a/website/static/img/docs/collaborate/dbt-explorer/example-prop-inherit.png and /dev/null differ diff --git a/website/static/img/docs/collaborate/dbt-explorer/explore-staging-env.png b/website/static/img/docs/collaborate/dbt-explorer/explore-staging-env.png index ef218f962e0..6cd5d5b379b 100644 Binary files a/website/static/img/docs/collaborate/dbt-explorer/explore-staging-env.png and b/website/static/img/docs/collaborate/dbt-explorer/explore-staging-env.png differ diff --git a/website/static/img/docs/collaborate/dbt-explorer/explorer-main-page.gif b/website/static/img/docs/collaborate/dbt-explorer/explorer-main-page.gif new file mode 100644 index 00000000000..6009e54197f Binary files /dev/null and b/website/static/img/docs/collaborate/dbt-explorer/explorer-main-page.gif differ diff --git a/website/static/img/docs/collaborate/dbt-explorer/explorer-model-performance.gif b/website/static/img/docs/collaborate/dbt-explorer/explorer-model-performance.gif new file mode 100644 index 00000000000..8fdc61be395 Binary files /dev/null and b/website/static/img/docs/collaborate/dbt-explorer/explorer-model-performance.gif differ diff --git a/website/static/img/docs/dbt-cloud/add-api-assignment.png b/website/static/img/docs/dbt-cloud/add-api-assignment.png new file mode 100644 index 00000000000..6a5cdec3cb6 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/add-api-assignment.png differ diff --git a/website/static/img/docs/dbt-cloud/add-api-rule.png b/website/static/img/docs/dbt-cloud/add-api-rule.png new file mode 100644 index 00000000000..dfd2cd620dd Binary files /dev/null and b/website/static/img/docs/dbt-cloud/add-api-rule.png differ diff --git a/website/static/img/docs/dbt-cloud/add-api-scope.png b/website/static/img/docs/dbt-cloud/add-api-scope.png new file mode 100644 index 00000000000..5f3d5ae3eea Binary files /dev/null and b/website/static/img/docs/dbt-cloud/add-api-scope.png differ diff --git a/website/static/img/docs/dbt-cloud/callback-uri.png b/website/static/img/docs/dbt-cloud/callback-uri.png new file mode 100644 index 00000000000..50e5490382a Binary files /dev/null and b/website/static/img/docs/dbt-cloud/callback-uri.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-legacy-model.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-legacy-model.png new file mode 100644 index 00000000000..55e18a151ee Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-legacy-model.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-list.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-list.png new file mode 100644 index 00000000000..dc19f6c9f51 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-list.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-new-model.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-new-model.png new file mode 100644 index 00000000000..1e006923f45 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-new-model.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-2.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-2.png new file mode 100644 index 00000000000..f37306fa1c1 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-2.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-3.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-3.png new file mode 100644 index 00000000000..be36f5e4abd Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-3.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-4.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-4.png new file mode 100644 index 00000000000..26ed3a28ac8 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-4.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-5.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-5.png new file mode 100644 index 00000000000..f5003430286 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout-5.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout.png b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout.png new file mode 100644 index 00000000000..29141c1ae34 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/connections-post-rollout.png differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-applications.jpg b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-applications.jpg new file mode 100644 index 00000000000..9c2208df667 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-applications.jpg differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-review-request.jpg b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-review-request.jpg new file mode 100644 index 00000000000..4dffe3359b8 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-review-request.jpg differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-settings.jpg b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-settings.jpg new file mode 100644 index 00000000000..516f7261ec2 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/github-settings.jpg differ diff --git a/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/repository-details-faq.jpg b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/repository-details-faq.jpg new file mode 100644 index 00000000000..ac5f460b453 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/cloud-configuring-dbt-cloud/repository-details-faq.jpg differ diff --git a/website/static/img/docs/dbt-cloud/configure-new-connection.png b/website/static/img/docs/dbt-cloud/configure-new-connection.png new file mode 100644 index 00000000000..53c50593505 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/configure-new-connection.png differ diff --git a/website/static/img/docs/dbt-cloud/configure-okta-app.png b/website/static/img/docs/dbt-cloud/configure-okta-app.png new file mode 100644 index 00000000000..48c9ba8ba63 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/configure-okta-app.png differ diff --git a/website/static/img/docs/dbt-cloud/configure-token-lifetime.png b/website/static/img/docs/dbt-cloud/configure-token-lifetime.png new file mode 100644 index 00000000000..b73552355b5 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/configure-token-lifetime.png differ diff --git a/website/static/img/docs/dbt-cloud/create-okta-api.png b/website/static/img/docs/dbt-cloud/create-okta-api.png new file mode 100644 index 00000000000..c20a47078d4 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/create-okta-api.png differ diff --git a/website/static/img/docs/dbt-cloud/create-okta-app.png b/website/static/img/docs/dbt-cloud/create-okta-app.png new file mode 100644 index 00000000000..18f225fcbfe Binary files /dev/null and b/website/static/img/docs/dbt-cloud/create-okta-app.png differ diff --git a/website/static/img/docs/dbt-cloud/environment-options.png b/website/static/img/docs/dbt-cloud/environment-options.png index 0d1a342e60b..a962e2f4b91 100644 Binary files a/website/static/img/docs/dbt-cloud/environment-options.png and b/website/static/img/docs/dbt-cloud/environment-options.png differ diff --git a/website/static/img/docs/dbt-cloud/explore-nav.jpg b/website/static/img/docs/dbt-cloud/explore-nav.jpg deleted file mode 100644 index c0d26e58c56..00000000000 Binary files a/website/static/img/docs/dbt-cloud/explore-nav.jpg and /dev/null differ diff --git a/website/static/img/docs/dbt-cloud/gather-authorization-token-endpoints.png b/website/static/img/docs/dbt-cloud/gather-authorization-token-endpoints.png new file mode 100644 index 00000000000..7bf7a67a6ef Binary files /dev/null and b/website/static/img/docs/dbt-cloud/gather-authorization-token-endpoints.png differ diff --git a/website/static/img/docs/dbt-cloud/gather-clientid-secret.png b/website/static/img/docs/dbt-cloud/gather-clientid-secret.png new file mode 100644 index 00000000000..703ef19a736 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/gather-clientid-secret.png differ diff --git a/website/static/img/docs/dbt-cloud/gather-uris.png b/website/static/img/docs/dbt-cloud/gather-uris.png new file mode 100644 index 00000000000..8207b846812 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/gather-uris.png differ diff --git a/website/static/img/docs/dbt-cloud/metadata-example.png b/website/static/img/docs/dbt-cloud/metadata-example.png new file mode 100644 index 00000000000..cfc3aae8e26 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/metadata-example.png differ diff --git a/website/static/img/docs/dbt-cloud/metadata-uri.png b/website/static/img/docs/dbt-cloud/metadata-uri.png new file mode 100644 index 00000000000..59c5f857828 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/metadata-uri.png differ diff --git a/website/static/img/docs/dbt-cloud/no-option.png b/website/static/img/docs/dbt-cloud/no-option.png index d540ad96e0f..dc955c26cd2 100644 Binary files a/website/static/img/docs/dbt-cloud/no-option.png and b/website/static/img/docs/dbt-cloud/no-option.png differ diff --git a/website/static/img/docs/dbt-cloud/select-oauth-config.png b/website/static/img/docs/dbt-cloud/select-oauth-config.png new file mode 100644 index 00000000000..d00e1eb7d80 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/select-oauth-config.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/hex_collections.png b/website/static/img/docs/dbt-cloud/semantic-layer/hex_collections.png new file mode 100644 index 00000000000..e9f78792a9b Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/hex_collections.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/hex_dbt_toggle.png b/website/static/img/docs/dbt-cloud/semantic-layer/hex_dbt_toggle.png new file mode 100644 index 00000000000..d5a57f170e2 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/hex_dbt_toggle.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/hex_duplicate.png b/website/static/img/docs/dbt-cloud/semantic-layer/hex_duplicate.png new file mode 100644 index 00000000000..2a55d9e6799 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/hex_duplicate.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/hex_make_sl_cell.png b/website/static/img/docs/dbt-cloud/semantic-layer/hex_make_sl_cell.png new file mode 100644 index 00000000000..0ab0d74fae3 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/hex_make_sl_cell.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/hex_new.png b/website/static/img/docs/dbt-cloud/semantic-layer/hex_new.png new file mode 100644 index 00000000000..2c111bd7aea Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/hex_new.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/hex_new_data_connection.png b/website/static/img/docs/dbt-cloud/semantic-layer/hex_new_data_connection.png new file mode 100644 index 00000000000..bf4d9bb74f3 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/hex_new_data_connection.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/hex_rename.png b/website/static/img/docs/dbt-cloud/semantic-layer/hex_rename.png new file mode 100644 index 00000000000..3495b1e03b5 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/hex_rename.png differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/sl-add-credential.jpg b/website/static/img/docs/dbt-cloud/semantic-layer/sl-add-credential.jpg new file mode 100644 index 00000000000..b2139da47b0 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/sl-add-credential.jpg differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/sl-create-service-token-page.jpg b/website/static/img/docs/dbt-cloud/semantic-layer/sl-create-service-token-page.jpg new file mode 100644 index 00000000000..8e288183be2 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/sl-create-service-token-page.jpg differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/sl-credential-created.jpg b/website/static/img/docs/dbt-cloud/semantic-layer/sl-credential-created.jpg new file mode 100644 index 00000000000..8c0081129fa Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/sl-credential-created.jpg differ diff --git a/website/static/img/docs/dbt-cloud/semantic-layer/welcome_to_hex.png b/website/static/img/docs/dbt-cloud/semantic-layer/welcome_to_hex.png new file mode 100644 index 00000000000..8d588484600 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/semantic-layer/welcome_to_hex.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png new file mode 100644 index 00000000000..8e02ba262fe Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var-targetname.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png new file mode 100644 index 00000000000..c4754e4bf60 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/Environment Variables/custom-schema-env-var.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png new file mode 100644 index 00000000000..8e02ba262fe Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/custom-schema-env-var-targetname.png differ diff --git a/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png b/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png new file mode 100644 index 00000000000..c4754e4bf60 Binary files /dev/null and b/website/static/img/docs/dbt-cloud/using-dbt-cloud/env-var-target-name.png differ diff --git a/website/vercel.json b/website/vercel.json index 0eb0ea516e9..18d70f9fee0 100644 --- a/website/vercel.json +++ b/website/vercel.json @@ -2,6 +2,16 @@ "cleanUrls": true, "trailingSlash": false, "redirects": [ + { + "source": "docs/cloud/secure/environment-permissions", + "destination": "docs/cloud/manage-access/environment-permissions", + "permanent": true + }, + { + "source": "docs/cloud/secure/environment-permissions-setup", + "destination": "docs/cloud/manage-access/environment-permissions-setup", + "permanent": true + }, { "source": "/docs/dbt-versions/core-upgrade/upgrading-to-dbt-utils-v1.0.md", "destination": "/docs/dbt-versions/core-upgrade/Older%20versions/upgrading-to-dbt-utils-v1.0.md",