From 440d317cb985ec9c84b628e80d96450321623e4a Mon Sep 17 00:00:00 2001 From: Willy Hardy Date: Fri, 12 Jul 2024 06:35:41 -0400 Subject: [PATCH 1/8] add detailed instructions (#5767) ## What are you changing in this pull request and why? Add detailed instructions. ## Checklist - [ x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ x] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [ x] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --------- Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/guides/mesh-qs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/guides/mesh-qs.md b/website/docs/guides/mesh-qs.md index 87fae1b58e0..fe581471da3 100644 --- a/website/docs/guides/mesh-qs.md +++ b/website/docs/guides/mesh-qs.md @@ -460,10 +460,10 @@ models: ### Set up model versions In this section, you will set up model versions by the Data Analytics team as they upgrade the `fct_orders` model while offering backward compatibility and a migration notice to the downstream Finance team. -1. Rename the existing model file from` models/core/fct_orders.sql` to `models/core/fct_orders_v1.sql`. +1. Rename the existing model file from `models/core/fct_orders.sql` to `models/core/fct_orders_v1.sql`. 2. Create a new file `models/core/fct_orders_v2.sql` and adjust the schema: - - Comment out `orders.status` - - Add a new field, `is_return` to indicate if an order was returned. + - Comment out `o.status` in the `final` CTE. + - Add a new field, `case when o.status = 'returned' then true else false end as is_return` to indicate if an order was returned. 3. Then, add the following to your `models/core/core.yml` file: - The `is_return` column - The two model `versions` From cbeeab8a8a6643aed33afe2f9cf1dcd71079a0b3 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Fri, 12 Jul 2024 12:10:41 -0400 Subject: [PATCH 2/8] Removes Azure log exception (#5649) ## What are you changing in this pull request and why? See: https://dbt-labs.slack.com/archives/C02NCQ9483C/p1718142981395639 Closes https://github.com/dbt-labs/docs.getdbt.com/issues/5647 ## Checklist - [ ] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." Adding or removing pages (delete if not applicable): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages - [ ] Add an entry for deleted pages in `website/vercel.json` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- website/docs/docs/cloud/manage-access/audit-log.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/website/docs/docs/cloud/manage-access/audit-log.md b/website/docs/docs/cloud/manage-access/audit-log.md index d94e147b501..0abf54ff991 100644 --- a/website/docs/docs/cloud/manage-access/audit-log.md +++ b/website/docs/docs/cloud/manage-access/audit-log.md @@ -173,10 +173,3 @@ You can use the audit log to export all historical audit results for security, c - **For events beyond 90 days** — Select **Export All**. The Account Admin will receive an email link to download a CSV file of all the events that occurred in your organization. - -### Azure Single-tenant - -For users deployed in [Azure single tenant](/docs/cloud/about-cloud/tenancy), while the **Export All** button isn't available, you can conveniently use specific APIs to access all events: - -- [Get recent audit log events CSV](/dbt-cloud/api-v3#/operations/Get%20Recent%20Audit%20Log%20Events%20CSV) — This API returns all events in a single CSV without pagination. -- [List recent audit log events](/dbt-cloud/api-v3#/operations/List%20Recent%20Audit%20Log%20Events) — This API returns a limited number of events at a time, which means you will need to paginate the results. From 00c713bda18fb5509b38c9364496818f63619cf1 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <107218380+nghi-ly@users.noreply.github.com> Date: Fri, 12 Jul 2024 13:32:32 -0700 Subject: [PATCH 3/8] Align pill with expandable header (#5772) ## What are you changing in this pull request and why? Vertically align pill with the expandable alt_header, per https://github.com/dbt-labs/docs.getdbt.com/pull/5588#pullrequestreview-2093911588 ## Checklist - [x] Test changes in localhost Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- website/src/components/expandable/styles.module.css | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/website/src/components/expandable/styles.module.css b/website/src/components/expandable/styles.module.css index 8c37036ad86..fc6f258286b 100644 --- a/website/src/components/expandable/styles.module.css +++ b/website/src/components/expandable/styles.module.css @@ -141,3 +141,8 @@ .expandableContainer { margin-bottom: 5px; /* Adjust this value as needed to create space */ } + +.headerText { + display: flex; + align-items: center; +} \ No newline at end of file From b65a57760b79784584c887d786ef74fdba558fa6 Mon Sep 17 00:00:00 2001 From: Jeremy Yeo Date: Mon, 15 Jul 2024 09:03:08 +0000 Subject: [PATCH 4/8] Update unit-tests.md (#5782) ## What are you changing in this pull request and why? Typo. https://dbt-labs.slack.com/archives/C02SRNY2EQ4/p1720994485348989?thread_ts=1720782919.428489&cid=C02SRNY2EQ4 ## Checklist - [x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --- website/docs/docs/build/unit-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/build/unit-tests.md b/website/docs/docs/build/unit-tests.md index dcd7e6d282d..3d86b8be0fc 100644 --- a/website/docs/docs/build/unit-tests.md +++ b/website/docs/docs/build/unit-tests.md @@ -51,7 +51,7 @@ You should unit test a model: dbt Labs strongly recommends only running unit tests in development or CI environments. Since the inputs of the unit tests are static, there's no need to use additional compute cycles running them in production. Use them in development for a test-driven approach and CI to ensure changes don't break them. -Use the [resource type](/reference/global-configs/resource-type) flag `--exclude-resource-type` or the `DBT_EXCLUDE_RESOURCE_TYPE` environment variable to exclude unit tests from your production builds and save compute. +Use the [resource type](/reference/global-configs/resource-type) flag `--exclude-resource-type` or the `DBT_EXCLUDE_RESOURCE_TYPES` environment variable to exclude unit tests from your production builds and save compute. ## Unit testing a model From b36eafd7a1a56f62b69b8d46e2c543cfbb35bd30 Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 15 Jul 2024 16:19:05 +0100 Subject: [PATCH 5/8] add job memory limit faq (#5778) this pr adds a job memory limit faq for support and users who encounter this error. refer to slack: https://dbt-labs.slack.com/archives/C02NCQ9483C/p1720498592885069 --- website/docs/docs/deploy/job-scheduler.md | 4 +++ .../faqs/Troubleshooting/job-memory-limits.md | 26 +++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 website/docs/faqs/Troubleshooting/job-memory-limits.md diff --git a/website/docs/docs/deploy/job-scheduler.md b/website/docs/docs/deploy/job-scheduler.md index 5bb2083d3da..7d45fddc3f6 100644 --- a/website/docs/docs/deploy/job-scheduler.md +++ b/website/docs/docs/deploy/job-scheduler.md @@ -102,6 +102,10 @@ Example of deactivation banner on job's page: +## FAQs + + + ## Related docs - [dbt Cloud architecture](/docs/cloud/about-cloud/architecture#dbt-cloud-features-architecture) - [Job commands](/docs/deploy/job-commands) diff --git a/website/docs/faqs/Troubleshooting/job-memory-limits.md b/website/docs/faqs/Troubleshooting/job-memory-limits.md new file mode 100644 index 00000000000..cfbecf9e25d --- /dev/null +++ b/website/docs/faqs/Troubleshooting/job-memory-limits.md @@ -0,0 +1,26 @@ +--- +title: "I'm receiving a 'This run exceeded your account's run memory limits' error in my failed job" +description: "Use incremental models or optimize queries for job failures due to exceeded memory limits." +sidebar_label: 'Job failures due to exceeded memory limits' +--- + +If you're receiving a `This run exceeded your account's run memory limits` error in your failed job, it means that the job exceeded the [memory limits](/docs/deploy/job-scheduler#job-memory) set for your account. All dbt Cloud accounts have a pod memory of 600Mib and memory limits are on a per run basis. They're typically influenced by the amount of result data that dbt has to ingest and process, which is small but can become bloated unexpectedly by project design choices. + +## Common reasons + +Some common reasons for higher memory usage are: + +- dbt run/build: Macros that capture large result sets from run query may not all be necessary and may be memory inefficient. +- dbt docs generate: Source or model schemas with large numbers of tables (even if those tables aren't all used by dbt) cause the ingest of very large results for catalog queries. + +## Resolution + +Try the following to resolve this: + +1. **Use incremental models**: Try using [incremental models](/docs/build/incremental-models-overview) to reduce the amount of data being processed in each run. Incremental models only process new or updated data, which can help reduce the memory usage of your jobs. +2. **Refactor your data model**: Review your data models to see if there are any opportunities to optimize or refactor them. For example, you can try to reduce the number of columns being selected, use `where` clauses to filter data early in the query or use `limit` clauses to reduce the amount of data being processed. + +If you've tried the earlier suggestions and are still experiencing failed job runs with this error about hitting the memory limits of your account, please [reach out to support](mailto:support@getdbt.com) and we can try increasing your account's memory. We're happy to help! + +## Additional resources +- [Blog post on how we shaved 90 mins off](https://docs.getdbt.com/blog/how-we-shaved-90-minutes-off-model) From 50bb30425704cc4928360ae1ee5447de5a002ebf Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Mon, 15 Jul 2024 15:00:25 -0400 Subject: [PATCH 6/8] Azure IPs (#5726) ## What are you changing in this pull request and why? Adding preliminary Azure info to the regions and IPs page ## Checklist - [ ] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." Adding or removing pages (delete if not applicable): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages - [ ] Add an entry for deleted pages in `website/vercel.json` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages --------- Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Co-authored-by: mirnawong1 --- website/docs/docs/cloud/about-cloud/regions-ip-addresses.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md b/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md index 82c1f277edb..23b6c8ca906 100644 --- a/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md +++ b/website/docs/docs/cloud/about-cloud/regions-ip-addresses.md @@ -14,6 +14,7 @@ dbt Cloud is [hosted](/docs/cloud/about-cloud/architecture) in multiple regions |--------|----------|------------|--------------|----------------|-----------|-----------------| | North America [^1] | AWS us-east-1 (N. Virginia) | **Multi-tenant:** cloud.getdbt.com
**Cell based:** ACCOUNT_PREFIX.us1.dbt.com | 52.45.144.63
54.81.134.249
52.22.161.231
52.3.77.232
3.214.191.130
34.233.79.135 | ✅ | ✅ | ✅ | | EMEA [^1] | AWS eu-central-1 (Frankfurt) | emea.dbt.com | 3.123.45.39
3.126.140.248
3.72.153.148 | ❌ | ❌ | ✅ | +| EMEA [^1] | Azure
North Europe (Ireland) | **Cell based:** ACCOUNT_PREFIX.eu2.dbt.com (beta invite only) | 20.13.190.192/26 | ❌ | ❌ | ✅ | | APAC [^1] | AWS ap-southeast-2 (Sydney)| au.dbt.com | 52.65.89.235
3.106.40.33
13.239.155.206
| ❌ | ❌ | ✅ | | Virtual Private dbt or Single tenant | Customized | Customized | Ask [Support](/community/resources/getting-help#dbt-cloud-support) for your IPs | ❌ | ❌ | ✅ | From f441dd2fa4485cc42652c2d50a473bb9d851795c Mon Sep 17 00:00:00 2001 From: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> Date: Mon, 15 Jul 2024 20:13:54 +0100 Subject: [PATCH 7/8] Update authentication.md (#5784) remove gsheets in pat support example, confirmed by @rpourzand (SL PM ) --- website/docs/docs/dbt-cloud-apis/authentication.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/dbt-cloud-apis/authentication.md b/website/docs/docs/dbt-cloud-apis/authentication.md index 4d7c4d4c06a..8729cc0641d 100644 --- a/website/docs/docs/dbt-cloud-apis/authentication.md +++ b/website/docs/docs/dbt-cloud-apis/authentication.md @@ -33,7 +33,7 @@ pagination_prev: null You should use service tokens broadly for any production workflow where you need a service account. You should use PATs only for developmental workflows _or_ dbt Cloud client workflows that require user context. The following examples show you when to use a personal access token (PAT) or a service token: -* **Connecting a partner integration to dbt Cloud** — Some examples include Hightouch, Datafold, a custom app you’ve created, etc. These types of integrations should use a service token instead of a PAT because service tokens give you visibility, and you can scope them to only what the integration needs and ensure the least privilege. We highly recommend switching to a service token if you’re using a user API key for these integrations today. +* **Connecting a partner integration to dbt Cloud** — Some examples include the [dbt Semantic Layer Google Sheets integration](/docs/cloud-integrations/avail-sl-integrations), Hightouch, Datafold, a custom app you’ve created, etc. These types of integrations should use a service token instead of a PAT because service tokens give you visibility, and you can scope them to only what the integration needs and ensure the least privilege. We highly recommend switching to a service token if you’re using a user API key for these integrations today. * **Production Terraform** — Use a service token since this is a production workflow and is acting as a service account and not a user account. -* **Cloud CLI and Semantic Layer Sheets Integration** — Use a PAT since both the dbt Cloud CLI and Semantic Layer Google Sheets integrations work within the context of a user (the user is making the requests and has to operate within the context of their user account). -* **Testing a custom script and staging Terraform or Postman** — We recommend using a PAT as this is a developmental workflow and is scoped to the user making the changes. When you push this script or Terraform into production, use a service token instead. \ No newline at end of file +* **Cloud CLI** — Use a PAT since the dbt Cloud CLI works within the context of a user (the user is making the requests and has to operate within the context of their user account). +* **Testing a custom script and staging Terraform or Postman** — We recommend using a PAT as this is a developmental workflow and is scoped to the user making the changes. When you push this script or Terraform into production, use a service token instead. From 09825cfcb4ad42abc28926bbdf84d97b2f32a98a Mon Sep 17 00:00:00 2001 From: rpourzand Date: Tue, 16 Jul 2024 09:02:24 +0100 Subject: [PATCH 8/8] Update service-tokens.md (#5786) I was browsing some docs and noticed this. Teams accounts can add semantic layer permissions as well. We have active team accounts using the SL ## What are you changing in this pull request and why? ## Checklist - [ ] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [ ] For [docs versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#about-versioning), review how to [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content). - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." Adding or removing pages (delete if not applicable): - [ ] Add/remove page in `website/sidebars.js` - [ ] Provide a unique filename for new pages - [ ] Add an entry for deleted pages in `website/vercel.json` - [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages --- website/docs/docs/dbt-cloud-apis/service-tokens.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/docs/dbt-cloud-apis/service-tokens.md b/website/docs/docs/dbt-cloud-apis/service-tokens.md index 615e8a73536..26b0e58caa2 100644 --- a/website/docs/docs/dbt-cloud-apis/service-tokens.md +++ b/website/docs/docs/dbt-cloud-apis/service-tokens.md @@ -17,7 +17,7 @@ Service account tokens enable you to securely authenticate with the dbt Cloud AP You can use service account tokens for system-level integrations that do not run on behalf of any one user. Assign any permission sets available in dbt Cloud to your service account token, which can vary slightly depending on your plan: * Enterprise plans can apply any permission sets available to service tokens. -* Team plans can apply Account Admin, Member, Job Admin, Read-Only, and Metadata permissions set to service tokens. +* Team plans can apply Account Admin, Member, Job Admin, Read-Only, Metadata, and Semantic Layer permissions set to service tokens. You can assign as many permission sets as needed to one token. For more on permissions sets, see "[Enterprise Permissions](/docs/cloud/manage-access/enterprise-permissions)."