+
+ {title}
+ {readingTime && {date} · {readingTime} minute read} ++ {createPostPreview(description, 140)} +
+ Read more +- `true`
- `false`
+ * *Example: Integrated Development Environment (IDE)* + +**DO NOT** use an acronym if its only used once. Please use the full word or phrase for its one-time use. + +### Latin abbreviations + +Avoid using Latin abbreviations. These terms are harder to localize/translate. + +Some common Latin abbreviations and other words to use instead: + +| Avoid | Use | Example | +|--------------------|------------|---------| +| i.e. | that is | Use incremental models when your dbt runs are becoming too slow (that is, don't start with incremental models) | +| e.g. |
- for example
- like
- Join both the dedicated #adapter-ecosystem channel in dbt Slack and the channel for your adapter's data store (for example, #db-sqlserver and #db-athena)
- Using Jinja in SQL provides a way to use control structures (like `if` statements and `for` loops) in your queries
- and more
- and so forth
- A continuous integration environment running pull requests in GitHub, GitLab, and more
- While reasonable defaults are provided for many such operations (like `create_schema`, `drop_schema`, `create_table`, and so forth), you might need to override one or more macros when building a new adapter
Analysts already know the data that they want.
+ +If the raw `Login` dataset below contains two different date (`Login_Date` and `Session_Date`) fields, a data engineer would be stuck. They can’t just guess because using the wrong date field will create an entirely different metric! So they must go back to the analyst for clarity on which field to use. We’ve just had a full loop with two hand-offs, and the data engineer hasn’t even begun building the model yet. + +| Login_Date | Session_Date | User_Id | +|------------|--------------|---------| +| 2022-08-01 | 2022-08-01 | 123 | +| 2022-08-01 | 2022-08-03 | 123 | +| 2022-08-04 | 2022-08-04 | 975 | +| 2022-08-04 | 2022-08-04 | NULL | + +In comparison, your new analytics engineer could quickly pivot based on the findings in the raw data. They can skip the endless loops of “discovery”, because they can react in the moment. When they see the two date fields for `Login_Date` and `Session_Date`, they can instantly apply their product knowledge, compare against examples, or reach out to their business contacts to diagnose the difference and select the correct date for the model. + +If the business need includes looking at `Active Users` by `Groups A, B, and C`, then that adds an extra layer of complexity. Without an analytics engineer, you’ll see extra loops and hand-offs to finalize all the business logic, how to handle NULL values, and even just the final format. + +**Model format returned by data engineering** + +| Date | Active Users Group A | Active Users Group B | Active Users Group C | +|------------|----------------------|----------------------|----------------------| +| 2022-08-01 | 34 | 60 | 61 | +| 2022-08-02 | 77 | 86 | 37 | +| 2022-08-03 | 71 | 9 | 6 | +| 2022-08-04 | 63 | 87 | 10 | + +**Model format needed for the BI tool** + +| Date | Group | Active Users | +|------------|---------|--------------------| +| 2022-08-01 | Group A | 34 | +| 2022-08-01 | Group B | 60 | +| 2022-08-01 | Group C | 61 | +| 2022-08-02 | Group A | 77 | +| 2022-08-02 | Group B | 86 | +| 2022-08-02 | Group C | 37 | + +## Loop 2: Reacting to unexpected results + +Analysts are the first (and sometimes only) line of defense to catch data quality issues. Once a dataset is aggregated down to a single number to answer a business question, it’s often impossible to know if there’s an incorrect filter or wrong set of logic. + +When a model is built for the first time, all parties often “don’t know what they don’t know”. As the analyst digs into the curated dataset, they will find updates that need to be made to the model. Hence, our second point: + +Analysts question everything.
+ +`Group A` for `Active Users` may be focused on `Participant` roles. The stakeholder had explained to the analyst that they want to exclude any `Admins` that will have a different experience in the product. + +| User_Id | Location | Role | Level | Zone | +|---------|------------|-------------|-------|------| +| 123 | California | Editor | AAA | 1 | +| 427 | Utah | Participant | ABA | 1 | +| 864 | Georgia | Admin | CCC | 3 | + +A data engineer working off of a “build list” will add a filter for `WHERE Role = 'Participant'`. During the data validation step, the analyst would discover that there is actually a third `Role` of `Editor` that no one was aware of. This would create a loop where the data engineer would have to edit the model to replace it with `WHERE Role != 'Admin'`. + +The analyst caught the issue because they have the appropriate context to validate the dataset. Analysts work directly with the stakeholder, so they’ll also be more familiar with company-specific and department-specific acronyms. + +**As we all know, data validation is an art, not a science.** Analysts will employ everything from a “sniff test” (view a random sample of rows) to individual examples (one-to-one exact matching to another system). An analyst will have to use their experience to know when the dataset is “good enough” for the stakeholder and their question since 100% accuracy might not be the goal. And if we're being honest, sometimes being directionally correct is all that’s needed to make a business decision. + +An analyst is able to identify which areas do *not* need to be 100% accurate, which means they can also identify which areas *do* need to be 100% accurate. + +> dbt makes it very quick to add [data quality tests](/docs/build/tests). In fact, it’s so quick, that it’ll take an analyst longer to write up what tests they want than it would take for an analyst to completely finish coding them. + +When data quality issues are identified by the business, we often see that analysts are the first ones to be asked: + +- Why are over half of the `User_Ids` now showing as NULL? +- Why is this chart showing a User’s `Location` that’s not in the USA? +- Why is the dashboard filter saying that all User’s `Zones` are Zone 2? + +This familiarity with what types of data quality issues are the most important to the business means that an analyst can often identify in advance which automated tests should be added to a data model. + +## Loop 3: Reacting to mismatched documentation + +There’s nothing worse than coming back to view a curated dataset months later (or maybe after someone on the team has offboarded) and learning that there is nothing written to explain why certain logic exists. Or worse, the documentation exists but it no longer matches what the model actually does. Which brings us to our third and final point: + +Analysts understand the pain of a poorly documented dataset.
+ +It’s much easier to keep to a naming guide when the writer has a deep understanding of the data and how it’s referenced by the business. Analysts are often asked repeat questions by stakeholders, which is extra incentive to create great documentation from the beginning. + +> A data engineer will document a dataset based on what they needed to know to build it. An analyst with analytics engineering skills will document a dataset based on *how to use it going forward.* + +If we want to know how certain logic was built technically, then we can reference the SQL code in dbt docs. If we want to know *why* a certain logic was built into that specific model, then that’s where we’d turn to the documentation. + +- Example of not-so-helpful documentation ([dbt docs can](https://docs.getdbt.com/docs/building-a-dbt-project/documentation) build this dynamically): + - `Case when Zone = 1 and Level like 'A%' then 'True' else 'False' end as GroupB` +- Example of better, more descriptive documentation (add to your dbt markdown file or column descriptions): + - Group B is defined as Users in Zone 1 with a Level beginning with the letter 'A'. These users are accessing our new add-on product that began in Beta in August 2022. It's recommended to filter them out of the main Active Users metric. + +In the second example, the documentation is written with the stakeholder and business in mind, and analysts are experts at this translation of the technical to the non-technical. + +## Are you convinced? + +Teach your analyst to be an analytics engineer, so they can do their own model building for curated datasets. You’ll start to see their process evolve into multiple smaller iterations, as they add fields and test them as they build. No loops, no hand-offs, and no Telephone Game. Your new workflow will look a little like this: + + + +Instead of trying to define all the various groups of `Active Users` at once, an analytics engineer can be validating the `Group A` rows, while adding `Group B` to their local environment, while still working with the stakeholder on the definition of `Group C`. + +**All this to say: Your analyst turned analytics engineer is the key to unlocking fast development of curated datasets with next-level data quality and documentation.** + +### How to empower your analysts to level up their skillset + +Here are some next steps to get you started on converting your analysts to analytics engineers: + +1. Many analysts are already very comfortable querying data using SQL Select statements. For those that aren’t, challenge them to start pulling data for ad-hoc requests using SQL. Ask them to compare some of the common transformations in your BI tool to SQL functions or rebuild them using
+
+
+**Is this what I should do?**
+
+[It depends!](https://twitter.com/SBinLondon/status/1413113782214266885) There will be times when it might be better to have logic stored in the modeling layer and there will be times when it might be better to have logic stored in the semantic layer. Our shorthand is to only include logic in the semantic layer if it is needed by our stakeholders - if they don't need to analyze the components, we keep them in the modeling layer. In the end, the needs of your business stakeholders should drive your decision on where to keep this logic.
+
+## How to structure your metrics
+
+Now that we’ve designed our metrics, let's move on to structuring them within our project. We'll examine the different ways to organize metrics and take a look at the pros and cons of several strategies.
+
+### Folder structure
+
+If you follow [dbt’s best practices for structuring your project](https://docs.getdbt.com/guides/best-practices/how-we-structure/1-guide-overview), you will have a folder structure that looks similar to this:
+
+```yaml
+models:
+ staging:
+ intermediate:
+ marts:
+```
+
+Your marts folder would most likely contain your end-state models ready for business consumption. Given that metrics are meant for business consumption, we are presented with two options - staying within the same framework or representing metrics as their own level.
+
+We recommend Option A (metrics within marts) but recognize that some people might prefer Option B (metrics within models).
+
+**A. Metrics within marts**
+
+Create a metrics folder within marts and use this to contain all of your metric definitions.
+
+```yaml
+models:
+ staging:
+ intermediate:
+ marts:
+ metrics:
+```
+
+**B. Metrics within models**
+
+Create a metrics folder within models and use this to contain all of your metric definitions.
+
+```yaml
+models:
+ staging:
+ intermediate:
+ marts:
+ metrics:
+```
+
+### File structure
+
+Once you’ve decided ***where*** to put your metrics folder, you can now decide ***how*** you want to structure your metrics within this folder. Choose one of two methods for structuring metrics:
+
+**Option A: The all-in-one YML method**
+This method follows a similar pattern to [dbt’s best practices around model structure](https://docs.getdbt.com/guides/best-practices/how-we-structure/1-guide-overview). The introduction of the metrics folder is the only change from the standard best practice.
+
+In practice, the all-in-one YML method would look like the following:
+
+```yaml
+## Metrics within Marts
+models:
+ marts:
+ metrics:
+ - metrics.yml
+------
+## Metrics within Models
+models:
+ metrics:
+ - metrics.yml
+```
+
+**Option B: The single-metric-per-file method**
+In this method, you create *one* yml file for *each* metric*.* Although this is an opinionated stance that differs from [dbt’s best practices](https://docs.getdbt.com/guides/best-practices/how-we-structure/1-guide-overview), here are some reasons why this **could** be useful:
+
+- Individual files are more easily discovered by new analytics engineers as your organization expands
+- Individual files can more easily define specific code owners that may not be part of the data team.
+
+For example, Jerry from the Finance department is the code owner for the `revenue` metric definition and oversees it for the business. So, any change to this specific file would need Jerry’s sign-off.
+
+This can be tricky for code owners who aren’t familiar with your git flow, but it brings them into the chain of responsibility for the metric definition. It also helps them take ownership for reporting on this metric and creates a responsible party when definitions need to change.
+
+The single-file-code-owner method would look like this:
+
+```yaml
+models:
+ metrics:
+ marts:
+ - revenue.yml
+ - average_order_value.yml
+ - some_other_metric_name.yml
+```
+
+### Folder and file structure is a preference, not a hard rule
+
+In the end, all of the structuring information above is just a recommendation. Your project probably has a defined convention in how nodes are organized, whether or not it follows dbt best practices, and you should continue to follow your own organizational practices. That said, we do recommend that metrics should be separate from model yml files. The reason?
+
+**Metrics are important business objects unto themselves and should live separate from the model definitions.**
+
+## A call to action
+
+This is just the beginning of dbt metrics and the Semantic Layer. We have a number of exciting ideas for expanding capabilities that we plan to begin work on in the coming months. However, we can’t do that without you.
+
+This semantic layer is a fundamental change to what it means to interact with dbt and ultimately most of the best practices will come from the dbt Community - folks like you. It does not matter if you consider yourself an "expert" on this - we want to talk to you and hear how you are using or would like to use metrics and the semantic layer. Y’all are going to be our guiding light to help us make sure that all the functionality we add helps **you** serve the needs of your business.
+
+If your experience with the Semantic Layer match what we’ve written in this post, and especially if they don’t, please share [comments and feedback in this Discourse Discussion](https://discourse.getdbt.com/t/how-to-design-and-structure-metrics/5040)!
+
+Additionally, I would invite you to join us over at #dbt-core-metrics on the dbt Slack where we’ll be posting updates, answering questions, discussing usage, and hopefully responding with the best emojis.
diff --git a/website/blog/2022-10-19-polyglot-dbt-python-dataframes-and-sql.md b/website/blog/2022-10-19-polyglot-dbt-python-dataframes-and-sql.md
new file mode 100644
index 00000000000..5a18890e906
--- /dev/null
+++ b/website/blog/2022-10-19-polyglot-dbt-python-dataframes-and-sql.md
@@ -0,0 +1,347 @@
+---
+title: "Stronger together: Python, dataframes, and SQL"
+description: "Going polyglot is a major next step in the journey of dbt Core. It expands possibilities and also the potential for confusion right along with it. SQL, dataframes, and Python are stronger together and polyglot dbt allows informed practitioners to choose when to use each."
+slug: polyglot-dbt-python-dataframes-sql
+
+authors: [doug_beatty]
+
+tags: [dbt product updates]
+hide_table_of_contents: false
+
+date: 2022-10-18
+is_featured: true
+---
+
+# Stronger together: Python, dataframes, and SQL
+
+For years working in data and analytics engineering roles, I treasured the daily camaraderie sharing a small office space with talented folks using a range of tools - from analysts using SQL and Excel to data scientists working in Python. I always sensed that there was so much we could work on in collaboration with each other - but siloed data and tooling made this much more difficult. The diversity of our tools and languages made the potential for collaboration all the more interesting, since we could have folks with different areas of expertise each bringing their unique spin to the project. But logistically, it just couldn’t be done in a scalable way.
+
+So I couldn’t be more excited about dbt’s polyglot capabilities arriving in dbt Core 1.3. This release brings Python dataframe libraries that are crucial to data scientists and enables general-purpose Python but still uses a shared database for reading and writing data sets. Analytics engineers and data scientists are stronger together, and I can’t wait to work side-by-side in the same repo with all my data scientist friends.
+
+Going polyglot is a major next step in the journey of dbt Core. While it expands possibilities, we also recognize the potential for confusion. When combined in an intentional manner, SQL, dataframes, and Python are also stronger together. Polyglot dbt allows informed practitioners to choose the language that best fits your use case.
+
+In this post, we’ll give you your hands-on experience and seed your imagination with potential applications. We’ll walk you through a [demo](https://github.com/dbt-labs/demo-python-blog) that showcases string parsing - one simple way that Python can be folded into a dbt project.
+
+We’ll also give you the intellectual resources to compare/contrast:
+- different dataframe implementations within different data platforms
+- dataframes vs. SQL
+
+Finally, we’ll share “gotchas” and best practices we’ve learned so far and invite you to participate in discovering the answers to outstanding questions we are still curious about ourselves.
+
+Based on our early experiences, we recommend that you:
+
+✅ **Do**: Use Python when it is better suited for the job – model training, using predictive models, matrix operations, exploratory data analysis (EDA), Python packages that can assist with complex transformations, and select other cases where Python is a more natural fit for the problem you are trying to solve.
+
+❌ **Don’t**: Use Python where the solution in SQL is just as direct. Although a pure Python dbt project is possible, we’d expect the most impactful projects to be a mixture of SQL and Python.
+
+
+
+## Polyglot dbt: An alloy of Python, dataframes, and SQL
+
+dbt Core 1.3 [gives you the ability](https://www.getdbt.com/blog/introducing-support-for-python/) to use Python models to materialize dataframes as tables within your dbt DAG. In combination, Python and SQL models form an alloy within a dbt project, yielding net new properties not found in isolation.
+
+This is a simultaneous two-part unlock. One, we get a general purpose programming language in Python. Two, we get access to special-purpose dataframe libraries for reading from / writing to the database.
+
+Some pretty cool functionality follows:
+1. **Vast ecosystem of Python packages**: The rich Python ecosystem of packages is the heart of varied use cases like machine learning, exploratory data analysis (EDA), data set generation, and many, many more.
+1. **Dataframe syntax for data set manipulation**: There’s a vibrant community of Python-first developers that can more easily contribute to dbt projects when they can use dataframe syntax. This is especially useful in conjunction with data science use-cases.
+1. **Python workflows where your data lives**: Most Python work being done today is done outside of the data platform that stores the source data, meaning developers have to transfer the data into the Python environment first, which adds significant friction to the development workflow. In the case of Snowpark DataFrames, data movement is reduced by pushing the computation down to the data platform.
+
+## Trade-offs between SQL and dataframe syntax
+
+Once you’re set up - the next question is: what _should_ you use Python for? How should you think about tradeoffs between SQL vs. dataframes? I haven’t personally pondered this deeply… but the folks at [Ponder](https://ponder.io/) have 😉 They published a series of posts comparing the two in terms of:
+
+- [convenience](https://ponder.io/pandas-vs-sql-part-4-pandas-is-more-convenient/) - dataframes go well with data science libraries like [Scikit-learn](https://scikit-learn.org/stable/)
+- [conciseness](https://ponder.io/pandas-vs-sql-part-2-pandas-is-more-concise/) - dataframes have concise syntax for normalization, one-hot encoding, rolling averages, and other uses
+- [flexibility](https://ponder.io/pandas-vs-sql-part-3-pandas-is-more-flexible/) - rows and columns in dataframes can be transposed and functions can be applied along columns or rows
+
+Gaining **your own** sense of these differences will empower you to create your own alloy mix of polyglot dbt models.
+
+## Comparing dataframe libraries
+
+Before we get into our hands-on example, let’s take a look at the nuts and bolts of getting your project working with different dataframe types. Multiple data platforms and dataframe libraries are supported in dbt Core as of version 1.3, but not uniformly (see compatibility table below). See [here](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/python-models#specific-data-platforms) for platform-specific setup instructions.
+
+| **Type of data frame** | **Snowflake** | **Databricks** | **BigQuery** |
+|----------------------------|------------------------------------|-----------------------------------|---------------|
+| Snowpark DataFrames | ✅ | ❌ | ❌ |
+| pandas DataFrames | ✅ | ✅ | ✅ |
+| Spark DataFrames | ❌ | ✅ | ✅ |
+| pandas-on-Spark DataFrames | ❌ | ✅ | ✅ |
+| Koalas DataFrames | ❌ | ✅ | ✅ |
+
+Why does this work?
+ +This is a slightly different way of calculating NPS from the usually provided formula but it ends up with the same result. Here is why: + +- `promoter_pct` was defined as `total_promoter_respondents` / `total_respondents` + - In our example dataset, this nets out to 3 / 6 = 50%. + - If we instead assign a value of 100 and take the average, it becomes 300 / 6 = 50. +- `detractor_pct` was defined as `total_detractor_respondents` / `total_respondents` + - In our example dataset, this nets out to 1 / 6 = 16.67%. + - If we instead assign a value of 100 and take the average, it becomes -100 / 6 = -16.67. +- Therefore, our `nps_score` follows suit: + - In our example dataset, 50% - 16.67% = 33.33% + - If we instead assign a value of 100 and take the average, it becomes 200 / 6 = 33.33 + +The underlying principle of why this works is based on the fact that averages divide the sum of the values in the set by their number. In more dbt friendly terms, what it really means is that average is creating the following equation: `sum(value)/count(*)`. In the first example implementation, we were doing roughly the same thing with multiple metric definitions - the only difference was our numerator was a count that assigned each row a value of 1. So if we duplicate that logic and give each row a value of 1 then we can create far fewer metrics. + +But that only gets us to the `promoter_pct` and `detractor_pct` metrics. In order to combine these both into a single metric definition, we needed to change the value that we assign. Given that the total range of values that the metric could output is -100 (all detractors) to 100 (all promoters) we can assign each of those categories that peak value, along with 0 for passives. This means that when the numerator is aggregated, it nets out promoters against detractors just like the documented equation does `promoter score - detractor score` . + +
+
+
+## First hands-on experience using a Python package
+
+Now that we have sufficient background covered, let’s dive into a real-world use-case.
+
+The full source code for this demo is available at:
+- https://github.com/dbt-labs/python-string-parsing
+
+This example is using [DuckDB](https://duckdb.org/) as the database backend. You can use this same code by copying it in your project that is using your database adapter of choice.
+
+Our example will give you hands-on experience with three things:
+1. read data from a table into a dataframe
+1. parse uniform datetimes out of a variety of string formats using the `dateutil` library
+1. writes the result back into a table
+
+Date/time parsing is a common use-case because dates and times are ubiquitous in transactional source data. Often, the source format is unknown, and it may even be a mixture of multiple formats. The `dateutil` package has a method that will do a best-effort attempt at parsing a string into a Python datetime object, and it will raise an exception when it is unable to parse the input. (Users of pandas may already be familiar with the `pandas.to_datetime` method, which uses `dateutil`). The following demo uses `dateutil` to parse source data with unknown date/time formats.
+
+In this example we’ll:
+1. Install the requirements in a virtual environment
+1. Build the dbt project
+1. Examine the output
+
+### Components of the dbt project
+
+This dbt project has only two main pieces (besides our seed data to mimic source data):
+1. Transformation logic within the dbt Python model
+1. Configuration of data testing for quality assurance
+
+#### The dbt Python model
+
+```python
+import dateutil
+
+
+def try_dateutil_parse(x):
+ try:
+ return dateutil.parser.parse(x)
+ except:
+ return
+
+
+def model(dbt, session):
+ df = dbt.ref("source_data")
+
+ df['parsed_transaction_time'] = df['transaction_time'].apply(try_dateutil_parse)
+
+ return df
+```
+
+This model tries to parse the raw string value into a Python datetime. When not successful, it yields a `None` value rather than raising an error. The `dateutil` can handle a wider range of formats than most data platforms’ native functionality.
+
+#### Testing the result
+
+During the build process, dbt will check if any of the values are null. This is using the built-in [`not_null`](https://docs.getdbt.com/docs/building-a-dbt-project/tests#generic-tests) test, which will generate and execute SQL in the data platform.
+
+Our initial recommendation for testing Python models is to use [generic](https://docs.getdbt.com/docs/building-a-dbt-project/tests#generic-tests) and [singular](https://docs.getdbt.com/docs/building-a-dbt-project/tests#singular-tests) tests.
+
+```yaml
+version: 2
+
+models:
+ - name: transactions
+ columns:
+ - name: parsed_transaction_time
+ tests:
+ - not_null
+```
+
+### Download the repository and install requirements
+
+The full source code for this demo is available at:
+- https://github.com/dbt-labs/python-string-parsing
+
+The only prerequisites for this demo are `python3` and `git`. You can verify both are installed and you’re on the right version via these commands in your terminal:
+```shell
+python3 --version
+git --version
+```
+
+Assuming both are availble in your system, then you can clone the example repository using your method of choice:
+
+More detailed comparisons and trade-offs
+ +Snowpark DataFrames are only supported in Snowflake, while Spark DataFrames are only available on Databricks and BigQuery. It’s also worth keeping in mind that different types of dataframes use [different syntax](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/python-models#dataframe-api-and-syntax). + +We’ve intentionally not included Jinja within Python models: a primary use of Jinja is for control flow and accessing dynamic-esque variables both of which you can just do within Python! The other main use for Jinja within dbt is for creating abstractions across differing database syntaxes. At this time, there’s no unified syntax abstraction across the different types of dataframes. (But someone will probably [make one](https://xkcd.com/927/)!) + +Although pandas DataFrames may seem like the ideal solution due to their support across data platforms, they come with their own tradeoffs. For instance, they can’t take advantage of the efficiency of native types like Spark and Snowpark DataFrames. They’re also limited by memory – large data sets will quickly exhaust available memory. In addition to that, they are constrained by being single-threaded, so they can not take advantage of multiple cores. Although pandas can be parallelized via solutions like Dask, Modin, etc., both Snowpark and Spark DataFrames will handle these scaling challenges natively. So use Snowpark, pandas-on-Spark DataFrames, and Spark DataFrames whenever you can! (Note: pandas-on-Spark DataFrames were formerly known as Koalas DataFrames and now commonly called pandas API DataFrames.) + +In Snowflake, any Snowpark DataFrame transformations specified in Python are actually compiled to SQL before execution. + +
+
+
+HTTPS
+ +```shell +git clone https://github.com/dbt-labs/demo-python-blog.git +cd demo-python-blog +``` + +
+
+
+SSH
+ +```shell +git clone git@github.com:dbt-labs/demo-python-blog.git +cd demo-python-blog +``` + +
+
+
+Then you'll create a virtual environment and install all the dependencies. Choose your shell / operating system from the list below and run the commands (defaulting to `zsh`/`bash`):
+
+GitHub CLI
+ +```shell +gh repo clone dbt-labs/demo-python-blog +cd demo-python-blog +``` + +
+
+
+POSIX bash/zsh
+ +```shell +python3 -m venv env +source env/bin/activate +python3 -m pip install --upgrade pip +python3 -m pip install -r requirements.txt +source env/bin/activate +``` + +
+
+
+POSIX fish
+ +```shell +python3 -m venv env +source env/bin/activate.fish +python3 -m pip install --upgrade pip +python3 -m pip install -r requirements.txt +source env/bin/activate.fish +``` +
+
+
+POSIX csh/tcsh
+ +```shell +python3 -m venv env +source env/bin/activate.csh +python3 -m pip install --upgrade pip +python3 -m pip install -r requirements.txt +source env/bin/activate.csh +``` +
+
+
+POSIX PowerShell Core
+ +```shell +python3 -m venv env +env/bin/Activate.ps1 +python3 -m pip install --upgrade pip +python3 -m pip install -r requirements.txt +env/bin/Activate.ps1 +``` +
+
+
+Windows cmd.exe
+ +```shell +python -m venv env +env\Scripts\activate.bat +python -m pip install --upgrade pip +python -m pip install -r requirements.txt +env\Scripts\activate.bat +``` +
+
+
+
+### Build it
+Once the dependencies are all installed, we can build the project:
+
+```shell
+dbt build
+```
+
+### Query the result of the dbt transformation
+
+Congrats on successfully running your first dbt Python model! Let’s confirm the output visually by running the following query:
+
+```shell
+duckcli demo.duckdb --table --execute "select id, transaction_time, parsed_transaction_time from parse_datetimes order by id"
+```
+
+As you can see, each of the various input formats were successfully parsed into a uniform and standardized format.
+
+| id | transaction_time | parsed_transaction_time |
+|----|----------------------------------|-------------------------|
+| 1 | Fri, 16 Dec 2022 02:59:36 +0000 | 2022-12-16 02:59:36 |
+| 2 | Sun, 25 Dec 22 02:59:36 +0000 | 2022-12-25 02:59:36 |
+| 3 | Thursday, 31-Mar-22 02:59:36 UTC | 2022-03-31 02:59:36 |
+| 4 | Thu, 02 Jun 22 02:59:36 +0000 | 2022-06-02 02:59:36 |
+| 5 | Thu, 17 Feb 2022 02:59:36 +0000 | 2022-02-17 02:59:36 |
+| 6 | 2022-03-28 02:59:36+00:00 | 2022-03-28 02:59:36 |
+| 7 | 2022-10-22 02:59:36+00:00 | 2022-10-22 02:59:36 |
+| 8 | 2022-10-02 02:59:36+00:00 | 2022-10-02 02:59:36 |
+| 9 | Monday, 03-Jan-2022 02:59:36 UTC | 2022-01-03 02:59:36 |
+| 10 | Thu, 25 Aug 2022 02:59:36 +0000 | 2022-08-25 02:59:36 |
+| 11 | 2022-05-29T02:59:36+00:00 | 2022-05-29 02:59:36 |
+| 12 | 2022-08-02 02:59:36+00:00 | 2022-08-02 02:59:36 |
+| 13 | 2022-10-18 02:59:36+00:00 | 2022-10-18 02:59:36 |
+
+⚠️ Remember to deactivate the environment as a final step:
+```
+deactivate
+```
+
+And there you go! A real live python example for a common analytics use case! Now let’s think about what we learned.
+
+## Best practice recommendations
+
+Based on our early experiences using dbt Python models and our ongoing conversations with members of the dbt Community, here are some recommended “do’s and don'ts” we think will help set you up for success.
+
+### ✅ **Do**:
+
+- Use Python when it is better suited for the job – model training, using predictive models, matrix operations, exploratory data analysis (EDA), and Python packages that can assist with complex transformations.
+- Use the native the dataframe type and syntax for your data platform. Use a notebook environment (and a small sample of your data set) for initial development before copying it into dbt.
+
+### ❌ **Don’t**:
+
+- Use Python where the solution in SQL is just as direct. Although a pure Python dbt project is possible, we’d expect the most impactful projects to be a mixture of SQL and Python.
+- Perform web scraping or download data from the web.
+- Use pandas unless absolutely necessary. Although pandas can be useful in the prototyping stage, scaling to larger data sets often requires a platform-native type like Snowpark, Spark, or pandas-on-Spark.
+- Translate your entire project to be Python-only. Although possible, if models are already written in SQL and there’s not a specific benefit to it being in Python, just leave it as SQL.
+
+### 🤷 We don’t know yet!
+There are several outstanding questions where you are invited to contribute to the discussion:
+- [Reusable Python logic](https://github.com/dbt-labs/dbt-core/discussions/5741)
+- [Debugging output](https://github.com/dbt-labs/dbt-core/discussions/5799)
+- [Testing Python models](https://github.com/dbt-labs/docs.getdbt.com/discussions/1811)
+- [Abstractions over different dataframe APIs](https://github.com/dbt-labs/dbt-core/discussions/5738)
+
+## Conclusion
+
+Python is particularly well suited for many use-cases within a dbt project, including:
+- **Data science** model training or in-line deployment
+- **Utilizing packages** within the Python ecosystem like [ftfy](https://ftfy.readthedocs.io/en/latest/), [dateutil](https://dateutil.readthedocs.io/en/stable/index.html), etc.
+- **Exploratory data analysis** (EDA) using functions and packages such as [`pandas.describe`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html), [Pandas profiling](https://pandas-profiling.ydata.ai/docs/master/index.html), [Great Expectations](https://github.com/great-expectations/great_expectations), etc.
+- Generating **synthetic data sets** using [Faker](https://faker.readthedocs.io/en/master/)
+- **Linear programming** libraries like [CVXOPT](https://cvxopt.org/), [PULP](https://github.com/coin-or/pulp), [CVXPY](https://www.cvxpy.org/), [ECOS](https://github.com/embotech/ecos-python), [Google OR-Tools](https://github.com/google/or-tools), [SciPy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linprog.html), etc.
+- **Simulation** using [SimPy](https://simpy.readthedocs.io/en/latest/examples/index.html), [Monte Carlo simulation](https://github.com/matsonj/nba-monte-carlo), what-if analysis
+- More yet to come!
+
+There is no one programming language to rule them all, so there’s no programming language hill that we are going to die on! We’ll make sure that dbt supports a wide variety of languages and let you make well-reasoned decisions for each individual use case.
+
+We are excited to hear what you discover and build! Please share with us in the [#dbt-core-python-models](https://getdbt.slack.com/archives/C03QUA7DWCW) or [#i-made-this](https://getdbt.slack.com/archives/C01NH3F2E05) channels in Slack.
diff --git a/website/blog/2022-10-24-demystifying-event-streams.md b/website/blog/2022-10-24-demystifying-event-streams.md
new file mode 100644
index 00000000000..39829c3bca0
--- /dev/null
+++ b/website/blog/2022-10-24-demystifying-event-streams.md
@@ -0,0 +1,277 @@
+---
+title: "Demystifying event streams: Transforming events into tables with dbt"
+description: "Pulling data directly out of application databases is commonplace in the MDS, but also risky. Apps change quickly, and application teams might update database schemas in unexpected ways, leading to pipeline failures, data quality issues, data delivery slow-downs. There is a better way. In this blog post, Charlie Summers (Merit) describes how their organization transforms application event streams into analytics-ready tables, more resilient to event scheme changes."
+slug: demystifying-event-streams
+
+authors: [charlie_summers]
+
+tags: [analytics craft]
+hide_table_of_contents: false
+
+date: 2022-11-04
+is_featured: true
+---
+
+Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a Windows PowerShell
+ +```shell +python -m venv env +env\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install -r requirements.txt +env\Scripts\Activate.ps1 +``` +- Every model can be traced back to a declared source in the dbt project (i.e. no "root" models).
- End-of-DAG fanout remains under a specified threshold. | +| Testing | - Every model has a
- The percentage of models that have minimum 1 test applied is greater than or equal to a specified threshold. | +| Documentation | - Every model has a [description](https://docs.getdbt.com/reference/resource-properties/description).
- The percentage of models that have a description is greater than or equal to a specified threshold. | +| Structure | - All models are named with the appropriate prefix aligned according to their model types (e.g. staging models are prefixed with `stg_`).
- The sql file for each model is in the subdirectory aligned with the model type (e.g. intermediate models are in an [intermediate subdirectory](https://docs.getdbt.com/guides/best-practices/how-we-structure/3-intermediate)).
- Each models subdirectory contains one .yml file that includes tests and documentation for all models within the given subdirectory. | +| Performance | - Every model that directly feeds into an [exposure](https://docs.getdbt.com/docs/build/exposures) is materialized as a
- No models are dependent on chains of "non-physically-materialized" models greater than a specified threshold. | + +For the full up-to-date list of covered rules, check out the package’s [README](https://github.com/dbt-labs/dbt-project-evaluator#rules-1), which outlines for each misalignment of a best practice: + +- Definition and clarifying example +- Reason for flagging the misalignment +- Any known exceptions to the rule +- How to remediate the issue + +There might be specific situations where you need to depart from our best practices. *That’s actually okay*, as long as you’ve reviewed the misalignment and made the active choice to do something different. We built this tool with simple mechanisms to customize the package behavior, including: + +- Disabling a package model to exclude a best practice from the entire evaluation process +- Overriding variables to adjust *how* a best practice is evaluated +- Documenting specific project exceptions to a best practice in a seed file + +For instructions and code snippets for each customization method, check out the [README](https://github.com/dbt-labs/dbt-project-evaluator#customization-1). + +## Try it out! + +To try out the package in your own project: + +1. **Install the package**: Check [dbt Hub](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest/) for the latest installation instructions, or read [the docs](https://docs.getdbt.com/docs/build/packages) for more information on installing packages. +2. **Run and test all of the models in the package**: Execute a `dbt build --select package:dbt_project_evaluator` command. +3. **Identify any warnings**: Each test warning indicates the presence of a type of misalignment. + +For *each warning* that pops up: + +1. Identify the model name. +2. Locate the related documentation in the package [README](https://github.com/dbt-labs/dbt-project-evaluator#rules-1). +3. Query the model to find the specific instances of the issue within your project. +4. Either fix the issue(s) or [customize](https://github.com/dbt-labs/dbt-project-evaluator#customization-1) the package to exclude the issue(s). + +In order to automatically maintain project quality as your team expands, you can enforce alignment with dbt Lab’s best practices on all future code changes by [adding this package as a CI check](https://github.com/dbt-labs/dbt-project-evaluator#running-this-package-as-a-ci-check-1). Every time one of your team members (or yourself) opens a PR, the CI check will automatically ensure that new code changes don’t introduce new misalignments. + +You can think of this as “linting” your dbt project to keep it aligned with our best practices — in the same way you might lint your SQL code to keep it aligned with your style guide. + +To add this package as a CI check: + +1. Override the severity of your tests using an [environment variable](https://docs.getdbt.com/docs/build/environment-variables). +2. Run this package as a step in your CI job. + +To watch a full demo of using this package in greater detail, make sure to check out [my Coalesce talk below](https://youtu.be/smbRwmcM1Ok) [demo starts at 7:35]. + +
+ +If something isn’t working quite right or you have ideas for future functionality, [open an issue in the Github repository](https://github.com/dbt-labs/dbt-project-evaluator/issues) or even contribute code of your own! + +Together, we can ensure that dbt projects across the galaxy are set up for success as they grow to infinity and beyond. + +
+
+
+
+
+
+
+
+
+
+```sql
+{{ config(
+ post_hook=[
+ 'grant `roles/bigquery.dataViewer` on {{ this.type }} {{ this }} to "user:someone@yourcompany.com"'
+ ]
+) }}
+
+select ...
+
+```
+
+
+
+
+
+
+
+
+
+```sql
+{{ config(
+ post_hook=[
+ "grant select on {{ this }} to `someone@yourcompany.com`"
+ ]
+) }}
+
+select ...
+
+```
+
+
+
+
+
+
+
+
+
+```sql
+{{ config(
+ post_hook=[
+ "grant select on {{ this }} to reporter"
+ ]
+) }}
+
+select ...
+
+```
+
+
+
+
+
+
+
+
+
+```sql
+{{ config(
+ post_hook=[
+ "grant select on {{ this }} to role reporter"
+ ]
+) }}
+
+select ...
+
+```
+
+
+
+
+
+
+
+
+
+```python
+import pyspark.sql.functions as F
+
+def model(dbt, session):
+ dbt.config(materialized = "incremental")
+ df = dbt.ref("upstream_table")
+
+ if dbt.is_incremental:
+
+ # only new rows compared to max in current table
+ max_from_this = f"select max(updated_at) from {dbt.this}"
+ df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0])
+
+ # or only rows from the past 3 days
+ df = df.filter(df.updated_at >= F.date_add(F.current_timestamp(), F.lit(-3)))
+
+ ...
+
+ return df
+```
+
+
+
+
+
+
+
+**Note:** Incremental models are supported on BigQuery/Dataproc for the `merge` incremental strategy. The `insert_overwrite` strategy is not yet supported.
+
+[`metric('some_metric)'`,
`metric('some_other_metric)'`]
`'month'`, `'quarter'`,
`'year'`, `'all_time'`
| The time grain that the metric will be aggregated to in the returned dataset | Required | +| dimensions | [`'plan'`,
`'country'`] | The dimensions you want the metric to be aggregated by in the returned dataset | Optional | +| secondary_calculations | [`metrics.period_over_period( comparison_strategy="ratio", interval=1, alias="pop_1wk")`] | Performs the specified secondary calculation on the metric results. Examples include period over period calculations, rolling calcultions, and period to date calculations. | Optional | +| start_date | `'2022-01-01'` | Limits the date range of data used in the metric calculation by not querying data before this date | Optional | +| end_date | `'2022-12-31'` | Limits the date range of data used in the metric claculation by not querying data after this date | Optional | +| where | `plan='paying_customer'` | A sql statment, or series of sql statements, that alter the **final** CTE in the generated sql. Most often used to limit the data to specific values of dimensions provided | Optional | + +#### Secondary Calculations +Secondary calculations are window functions you can add to the metric calculation and perform on the primary metric or metrics. + +You can use them to compare values to an earlier period, calculate year-to-date sums, and return rolling averages. You can add custom secondary calculations into dbt projects - for more information on this, reference the [package README](https://github.com/dbt-labs/dbt_metrics#secondary-calculations). + +### Developing metrics with `metrics.develop` + +
+
+| YAML key | Value description |
+| :--- | :--- |
+| [name](/reference/project-configs/name) | Your project’s name in [snake case](https://en.wikipedia.org/wiki/Snake_case) |
+| [version](/reference/project-configs/version) | Version of your project |
+| [require-dbt-version](/reference/project-configs/require-dbt-version) | Restrict your project to only work with a range of [dbt Core versions](/docs/dbt-versions/core) |
+| [profile](/reference/project-configs/profile) | The profile dbt uses to connect to your data platform |
+| [model-paths](/reference/project-configs/model-paths) | Directories to where your model and source files live |
+| [seed-paths](/reference/project-configs/seed-paths) | Directories to where your seed files live |
+| [test-paths](/reference/project-configs/test-paths) | Directories to where your test files live |
+| [analysis-paths](/reference/project-configs/analysis-paths) | Directories to where your analyses live |
+| [macro-paths](/reference/project-configs/macro-paths) | Directories to where your macros live |
+| [snapshot-paths](/reference/project-configs/snapshot-paths) | Directories to where your snapshots live |
+| [docs-paths](/reference/project-configs/docs-paths) | Directories to where your docs blocks live |
+| [vars](/docs/build/project-variables) | Project variables you want to use for data compilation |
+
+
+
+For complete details on project configurations, see [dbt_project.yml](/reference/dbt_project.yml).
+
+## New projects
+
+You can create new projects and [share them](/docs/collaborate/git-version-control) with other people by making them available on a hosted git repository like GitHub, GitLab, and BitBucket.
+
+After you set up a connection with your data platform, you can [initialize your new project in dbt Cloud](/docs/get-started/getting-started/set-up-dbt-cloud) and start developing. Or, run [dbt init from the command line](/reference/commands/init) to set up your new project.
+
+During project initialization, dbt creates sample model files in your project directory to help you start developing quickly.
+
+## Sample projects
+
+If you want to explore dbt projects more in-depth, you can clone dbt Lab’s [Jaffle shop](https://github.com/dbt-labs/jaffle_shop) on GitHub. It's a runnable project that contains sample configurations and helpful notes.
+
+If you want to see what a mature, production project looks like, check out the [GitLab Data Team public repo](https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt).
+
+## Related docs
+- [Best practices: How we structure our dbt projects](/guides/best-practices/how-we-structure/1-guide-overview)
+* [Get started with dbt Cloud](/docs/get-started/getting-started/set-up-dbt-cloud)
+* [Get started with dbt Core](/docs/get-started/getting-started-dbt-core)
diff --git a/website/docs/docs/build/python-models.md b/website/docs/docs/build/python-models.md
new file mode 100644
index 00000000000..3f213f1fdca
--- /dev/null
+++ b/website/docs/docs/build/python-models.md
@@ -0,0 +1,648 @@
+---
+title: "Python models"
+id: "python-models"
+---
+- [Overview](#overview)
+- [Configuring Python Models](#configuring-python-models)
+- [Python Specific Functionality](#configuring-python-models)
+- [Limitations](#limitations)
+- [Supported data platforms](#supported-data-platforms)
+
+dbt Core v1.3 adds support for Python models. Note that only [specific data platforms](#specific-data-platforms) support dbt-py models.
+
+We encourage you to:
+- Read [the original discussion](https://github.com/dbt-labs/dbt-core/discussions/5261) that proposed this feature.
+- Contribute to [best practices for developing Python models in dbt](https://github.com/dbt-labs/docs.getdbt.com/discussions/1811).
+- Weigh in on [next steps for Python models, beyond v1.3](https://github.com/dbt-labs/dbt-core/discussions/5742).
+- Join the **#beta-feedback-python-models** channel in the [dbt Community Slack](https://www.getdbt.com/community/join-the-community/).
+
+
+
+
+
+```python
+import holidays
+
+def is_holiday(date_col):
+ # Chez Jaffle
+ french_holidays = holidays.France()
+ is_holiday = (date_col in french_holidays)
+ return is_holiday
+
+def model(dbt, session):
+ dbt.config(
+ materialized = "table",
+ packages = ["holidays"]
+ )
+
+ orders_df = dbt.ref("stg_orders")
+
+ df = orders_df.to_pandas()
+
+ # apply our function
+ # (columns need to be in uppercase on Snowpark)
+ df["IS_HOLIDAY"] = df["ORDER_DATE"].apply(is_holiday)
+
+ # return final dataset (Pandas DataFrame)
+ return df
+```
+
+
+
+
+
+
+
+
+
+```python
+import holidays
+
+def is_holiday(date_col):
+ # Chez Jaffle
+ french_holidays = holidays.France()
+ is_holiday = (date_col in french_holidays)
+ return is_holiday
+
+def model(dbt, session):
+ dbt.config(
+ materialized = "table",
+ packages = ["holidays"]
+ )
+
+ orders_df = dbt.ref("stg_orders")
+
+ df = orders_df.to_pandas_on_spark() # Spark 3.2+
+ # df = orders_df.toPandas() in earlier versions
+
+ # apply our function
+ df["is_holiday"] = df["order_date"].apply(is_holiday)
+
+ # convert back to PySpark
+ df = df.to_spark() # Spark 3.2+
+ # df = session.createDataFrame(df) in earlier versions
+
+ # return final dataset (PySpark DataFrame)
+ return df
+```
+
+
+
+
+
+
+
+
+
+```python
+import snowflake.snowpark.types as T
+import snowflake.snowpark.functions as F
+import numpy
+
+def register_udf_add_random():
+ add_random = F.udf(
+ # use 'lambda' syntax, for simple functional behavior
+ lambda x: x + numpy.random.normal(),
+ return_type=T.FloatType(),
+ input_types=[T.FloatType()]
+ )
+ return add_random
+
+def model(dbt, session):
+
+ dbt.config(
+ materialized = "table",
+ packages = ["numpy"]
+ )
+
+ temps_df = dbt.ref("temperatures")
+
+ add_random = register_udf_add_random()
+
+ # warm things up, who knows by how much
+ df = temps_df.withColumn("degree_plus_random", add_random("degree"))
+ return df
+```
+
+
+
+**Note:** Due to a Snowpark limitation, it is not currently possible to register complex named UDFs within stored procedures and, therefore, dbt Python models. We are looking to add native support for Python UDFs as a project/DAG resource type in a future release. For the time being, if you want to create a "vectorized" Python UDF via the Batch API, we recommend either:
+- Writing [`create function`](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-batch.html) inside a SQL macro, to run as a hook or run-operation
+- [Registering from a staged file](https://docs.snowflake.com/ko/developer-guide/snowpark/reference/python/_autosummary/snowflake.snowpark.udf.html#snowflake.snowpark.udf.UDFRegistration.register_from_file) within your Python model code
+
+
+
+
+
+
+
+```python
+from pyspark.sql.types as T
+import pyspark.sql.functions as F
+import numpy
+
+# use a 'decorator' for more readable code
+@F.udf(returnType=T.DoubleType())
+def add_random(x):
+ random_number = numpy.random.normal()
+ return x + random_number
+
+def model(dbt, session):
+ dbt.config(
+ materialized = "table",
+ packages = ["numpy"]
+ )
+
+ temps_df = dbt.ref("temperatures")
+
+ # warm things up, who knows by how much
+ df = temps_df.withColumn("degree_plus_random", add_random("degree"))
+ return df
+```
+
+
+
+
+
+
+
+**Additional setup:** Snowpark Python is in Public Preview - Open and enabled by default for all accounts. You will need to [acknowledge and accept Snowflake Third Party Terms](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages.html#getting-started) to use Anaconda packages.
+
+**Installing packages:** Snowpark supports several popular packages via Anaconda. The complete list is at https://repo.anaconda.com/pkgs/snowflake/. Packages are installed at the time your model is being run. Different models can have different package dependencies. If you are using third-party packages, Snowflake recommends using a dedicated virtual warehouse for best performance rather than one with many concurrent users.
+
+**About "sprocs":** dbt submits Python models to run as "stored procedures," which some people call "sprocs" for short. By default, dbt will create a named sproc containing your model's compiled Python code, and then "call" it to execute. Snowpark has a Private Preview feature for "temporary" or "anonymous" stored procedures ([docs](https://docs.snowflake.com/en/LIMITEDACCESS/call-with.html)), which are faster and leave a cleaner query history. If this feature is enabled for your account, you can switch it on for your models by configuring `use_anonymous_sproc: True`. We plan to switch this on for all dbt + Snowpark Python models in a future release.
+
+
+
+```yml
+# I asked Snowflake Support to enable this Private Preview feature,
+# and now my dbt-py models run even faster!
+models:
+ use_anonymous_sproc: True
+```
+
+
+
+**Docs:** ["Developer Guide: Snowpark Python"](https://docs.snowflake.com/en/developer-guide/snowpark/python/index.html)
+
+
+
+
+
+**Submission methods:** Databricks supports a few different mechanisms to submit PySpark code, each with relative advantages. Some are better for supporting iterative development, while others are better for supporting lower-cost production deployments. The options are:
+- `all_purpose_cluster` (default): dbt will run your Python model using the cluster ID configured as `cluster` in your connection profile or for this specific model. These clusters are more expensive but also much more responsive. We recommend using an interactive all-purpose cluster for quicker iteration in development.
+ - `create_notebook: True`: dbt will upload your model's compiled PySpark code to a notebook in the namespace `/Shared/dbt_python_model/{schema}`, where `{schema}` is the configured schema for the model, and execute that notebook to run using the all-purpose cluster. The appeal of this approach is that you can easily open the notebook in the Databricks UI for debugging or fine-tuning right after running your model. Remember to copy any changes into your dbt `.py` model code before re-running.
+ - `create_notebook: False` (default): dbt will use the [Command API](https://docs.databricks.com/dev-tools/api/1.2/index.html#run-a-command), which is slightly faster.
+- `job_cluster`: dbt will upload your model's compiled PySpark code to a notebook in the namespace `/Shared/dbt_python_model/{schema}`, where `{schema}` is the configured schema for the model, and execute that notebook to run using a short-lived jobs cluster. For each Python model, Databricks will need to spin up the cluster, execute the model's PySpark transformation, and then spin down the cluster. As such, job clusters take longer before and after model execution, but they're also less expensive, so we recommend these for longer-running Python models in production. To use the `job_cluster` submission method, your model must be configured with `job_cluster_config`, which defines key-value properties for `new_cluster`, as defined in the [JobRunsSubmit API](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsSubmit).
+
+You can configure each model's `submission_method` in all the standard ways you supply configuration:
+
+```python
+def model(dbt, session):
+ dbt.config(
+ submission_method="all_purpose_cluster",
+ create_notebook=True,
+ cluster_id="abcd-1234-wxyz"
+ )
+ ...
+```
+```yml
+version: 2
+models:
+ - name: my_python_model
+ config:
+ submission_method: job_cluster
+ job_cluster_config:
+ spark_version: ...
+ node_type_id: ...
+```
+```yml
+# dbt_project.yml
+models:
+ project_name:
+ subfolder:
+ # set defaults for all .py models defined in this subfolder
+ +submission_method: all_purpose_cluster
+ +create_notebook: False
+ +cluster_id: abcd-1234-wxyz
+```
+
+If not configured, `dbt-spark` will use the built-in defaults: the all-purpose cluster (based on `cluster` in your connection profile) without creating a notebook. The `dbt-databricks` adapter will default to the cluster configured in `http_path`. We encourage explicitly configuring the clusters for Python models in Databricks projects.
+
+**Installing packages:** When using all-purpose clusters, we recommend installing packages which you will be using to run your Python models.
+
+**Docs:**
+- [PySpark DataFrame syntax](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.html)
+- [Databricks: Introduction to DataFrames - Python](https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html)
+
+
+
+
+
+The `dbt-bigquery` adapter uses a service called Dataproc to submit your Python models as PySpark jobs. That Python/PySpark code will read from your tables and views in BigQuery, perform all computation in Dataproc, and write the final result back to BigQuery.
+
+**Submission methods.** Dataproc supports two submission methods: `serverless` and `cluster`. Dataproc Serverless does not require a ready cluster, which saves on hassle and cost—but it is slower to start up, and much more limited in terms of available configuration. For example, Dataproc Serverless supports only a small set of Python packages, though it does include `pandas`, `numpy`, and `scikit-learn`. (See the full list [here](https://cloud.google.com/dataproc-serverless/docs/guides/custom-containers#example_custom_container_image_build), under "The following packages are installed in the default image"). Whereas, by creating a Dataproc Cluster in advance, you can fine-tune the cluster's configuration, install any PyPI packages you want, and benefit from faster, more responsive runtimes.
+
+Use the `cluster` submission method with dedicated Dataproc clusters you or your organization manage. Use the `serverless` submission method to avoid managing a Spark cluster. The latter may be quicker for getting started, but both are valid for production.
+
+**Additional setup:**
+- Create or use an existing [Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets)
+- Enable Dataproc APIs for your project + region
+- If using the `cluster` submission method: Create or use an existing [Dataproc cluster](https://cloud.google.com/dataproc/docs/guides/create-cluster) with the [Spark BigQuery connector initialization action](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/connectors#bigquery-connectors). (Google recommends copying the action into your own Cloud Storage bucket, rather than using the example version shown in the screenshot)
+
+
+
+
+
Details for the timestamp strategy
-Snapshot query results at `2019-01-01 11:00`: +Snapshot query results at `2019-01-01 11:00`
| id | status | updated_at | | -- | ------- | ---------------- | @@ -376,8 +378,9 @@ Snapshot results (note that `11:30` is not used anywhere): For the `check` strategy, the current timestamp is used to populate each column
+
@@ -84,7 +85,7 @@ You can also:
- Add tests to sources
- Add descriptions to sources, that get rendered as part of your documentation site
-These should be familiar concepts if you've already added tests and descriptions to your models (if not check out the guides on [testing](building-a-dbt-project/tests) and [documentation](documentation)).
+These should be familiar concepts if you've already added tests and descriptions to your models (if not check out the guides on [testing](/docs/build/tests) and [documentation](documentation)).
@@ -117,11 +118,11 @@ sources:
You can find more details on the available properties for sources in the [reference section](source-properties).
### FAQs
-
+
+Starting in v1.3, dbt Core adds support for **Python models**.
+
+dbt's Python capabilities are an extension of its capabilities with SQL models. If you're new to dbt, we recommend that you read this page first, before reading: ["Python Models"](python-models)
+
+
+
+A SQL model is a `select` statement. Models are defined in `.sql` files (typically in your `models` directory):
- Each `.sql` file contains one model / `select` statement
-- The name of the file is used as the model name
+- The model name is inherited from the filename.
- Models can be nested in subdirectories within the `models` directory
-When you execute the [`dbt run` command](run), dbt will build this model in your data warehouse by wrapping it in a `create view as` or `create table as` statement.
+When you execute the [`dbt run` command](run), dbt will build this model
-When you execute `dbt run`, dbt will build this as a views
-* build models in a target schema you define
-* use your file name as the view or views
+* Build models in a target schema you define
+* Use your file name as the view or
-Importantly, configurations are applied hierarchically — a configuration applied to a subdirectory will override any general configurations.
+It is important to note that configurations are applied hierarchically — a configuration applied to a subdirectory will override any general configurations.
You can learn more about configurations in the [reference docs](model-configs).
### FAQs
-
* `v0.20.0`: Both types of tests return a set of failing records. Previously, generic/schema tests returned a numeric value representing failures. Generic tests (f.k.a. schema tests) are defined using `test` blocks instead of macros prefixed `test_`.
@@ -27,7 +30,7 @@ There are two ways of defining tests in dbt:
Defining tests is a great way to confirm that your code is working correctly, and helps prevent regressions when your code changes. Because you can use them over and over again, making similar assertions with minor variations, generic tests tend to be much more common—they should make up the bulk of your dbt testing suite. That said, both ways of defining tests have their time and place.
:::tip Creating your first tests
-If you're new to dbt, we recommend that you check out our [Getting Started Tutorial](tutorial/getting-started.md) to build your first dbt project with models and tests.
+If you're new to dbt, we recommend that you check out our [Getting Started guide](/docs/get-started/getting-started/overview) to build your first dbt project with models and tests.
:::
## Singular tests
@@ -112,7 +115,7 @@ You can find more information about these tests, and additional configurations (
Those four tests are enough to get you started. You'll quickly find you want to use a wider variety of tests—a good thing! You can also install generic tests from a package, or write your own, to use (and reuse) across your dbt project. Check out the [guide on custom generic tests](custom-generic-tests) for more information.
:::info
-There are generic tests defined in some open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](package-management) to learn more!
+There are generic tests defined in some open source packages, such as [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) and [dbt-expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest/) — skip ahead to the docs on [packages](/docs/build/packages) to learn more!
:::
### Example
@@ -236,7 +239,6 @@ where {{ column_name }} is null
-
## Storing test failures
@@ -252,22 +254,16 @@ This workflow allows you to query and examine failing records much more quickly
+
+Python models are supported in dbt Core 1.3 and above. Learn more about [upgrading your version in dbt Cloud](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/cloud-upgrading-dbt-versions) and [upgrading dbt Core versions](https://docs.getdbt.com/docs/core-versions#upgrading-to-new-patch-versions).
+
+To read more about Python models, change the docs version to 1.3 or higher in the menu above.
+
+
+
+
+
+
+
+
+```python
+import ...
+
+def model(dbt, session):
+
+ my_sql_model_df = dbt.ref("my_sql_model")
+
+ final_df = ... # stuff you can't write in SQL!
+
+ return final_df
+```
+
+
+
+
+
+```yml
+version: 2
+
+models:
+ - name: my_python_model
+
+ # Document within the same codebase
+ description: My transformation written in Python
+
+ # Configure in ways that feel intuitive and familiar
+ config:
+ materialized: table
+ tags: ['python']
+
+ # Test the results of my Python transformation
+ columns:
+ - name: id
+ # Standard validation for 'grain' of Python results
+ tests:
+ - unique
+ - not_null
+ tests:
+ # Write your own validation logic (in SQL) for Python results
+ - [custom_generic_test](writing-custom-generic-tests)
+```
+
+
+
+
+DataFrame operations define the starting points, the end state, and each step along the way.
+
+This is similar to the role of CTEs in dbt SQL models. We use CTEs to pull in upstream datasets, define (and name) a series of meaningful transformations, and end with a final `select` statement. You can run the compiled version of a dbt SQL model to see the data included in the resulting view or table. When you `dbt run`, dbt wraps that query in `create view`, `create table`, or more complex DDL to save its results in the database.
+
+Instead of a final `select` statement, each Python model returns a final DataFrame. Each DataFrame operation is "lazily evaluated." In development, you can preview its data, using methods like `.show()` or `.head()`. When you run a Python model, the full result of the final DataFrame will be saved as a table in your data warehouse.
+
+dbt Python models have access to almost all of the same configuration options as SQL models. You can test them, document them, add `tags` and `meta` properties to them, grant access to their results to other users, and so on. You can select them by their name, their file path, their configurations, whether they are upstream or downstream of another model, or whether they have been modified compared to a previous project state.
+
+### Defining a Python model
+
+Each Python model lives in a `.py` file in your `models/` folder. It defines a function named **`model()`**, which takes two parameters:
+- **`dbt`**: A class compiled by dbt Core, unique to each model, enables you to run your Python code in the context of your dbt project and DAG.
+- **`session`**: A class representing your data platform’s connection to the Python backend. The session is needed to read in tables as DataFrames, and to write DataFrames back to tables. In PySpark, by convention, the `SparkSession` is named `spark`, and available globally. For consistency across platforms, we always pass it into the `model` function as an explicit argument called `session`.
+
+The `model()` function must return a single DataFrame. On Snowpark (Snowflake), this can be a Snowpark or pandas DataFrame. Via PySpark (Databricks + BigQuery), this can be a Spark, pandas, or pandas-on-Spark DataFrame. For more about choosing between pandas and native DataFrames, see [DataFrame API + syntax](#dataframe-api--syntax).
+
+When you `dbt run --select python_model`, dbt will prepare and pass in both arguments (`dbt` and `session`). All you have to do is define the function. This is how every single Python model should look:
+
+
+
+```python
+def model(dbt, session):
+
+ ...
+
+ return final_df
+```
+
+
+
+
+### Referencing other models
+
+Python models participate fully in dbt's directed acyclic graph (DAG) of transformations. Use the `dbt.ref()` method within a Python model to read in data from other models (SQL or Python). If you want to read directly from a raw source table, use `dbt.source()`. These methods return DataFrames pointing to the upstream source, model, seed, or snapshot.
+
+
+
+```python
+def model(dbt, session):
+
+ # DataFrame representing an upstream model
+ upstream_model = dbt.ref("upstream_model_name")
+
+ # DataFrame representing an upstream source
+ upstream_source = dbt.source("upstream_source_name", "table_name")
+
+ ...
+```
+
+
+
+Of course, you can `ref()` your Python model in downstream SQL models, too:
+
+
+
+```sql
+with upstream_python_model as (
+
+ select * from {{ ref('my_python_model') }}
+
+),
+
+...
+```
+
+
+
+### Configuring Python models
+
+Just like SQL models, there are three ways to configure Python models:
+1. In `dbt_project.yml`, where you can configure many models at once
+2. In a dedicated `.yml` file, within the `models/` directory
+3. Within the model's `.py` file, using the `dbt.config()` method
+
+Calling the `dbt.config()` method will set configurations for your model right within your `.py` file, similar to the `{{ config() }}` macro in `.sql` model files:
+
+
+
+```python
+def model(dbt, session):
+
+ # setting configuration
+ dbt.config(materialized="table")
+```
+
+
+
+There's a limit to how fancy you can get with the `dbt.config()` method. It accepts _only_ literal values (strings, booleans, and numeric types). Passing another function or a more complex data structure is not possible. The reason is that dbt statically analyzes the arguments to `config()` while parsing your model without executing your Python code. If you need to set a more complex configuration, we recommend you define it using the [`config` property](resource-properties/config) in a yaml file.
+
+#### Accessing project context
+
+dbt Python models don't use Jinja to render compiled code. Python models have limited access to global project contexts compared to SQL models. That context is made available from the `dbt` class, passed in as an argument to the `model()` function.
+
+Out of the box, the `dbt` class supports:
+- Returning DataFrames referencing the locations of other resources: `dbt.ref()` + `dbt.source()`
+- Accessing the database location of the current model: `dbt.this()` (also: `dbt.this.database`, `.schema`, `.identifier`)
+- Determining if the current model's run is incremental: `dbt.is_incremental`
+
+It is possible to extend this context by "getting" them via `dbt.config.get()` after they are configured in the [model's config](/reference/model-configs). This includes inputs such as `var`, `env_var`, and `target`. If you want to use those values to power conditional logic in your model, we require setting them through a dedicated `.yml` file config:
+
+
+
+```yml
+version: 2
+
+models:
+ - name: my_python_model
+ config:
+ materialized: table
+ target_name: "{{ target.name }}"
+ specific_var: "{{ var('SPECIFIC_VAR') }}"
+ specific_env_var: "{{ env_var('SPECIFIC_ENV_VAR') }}"
+```
+
+
+
+Then, within the model's Python code, use the `dbt.config.get()` function to _access_ values of configurations that have been set:
+
+
+
+```python
+def model(dbt, session):
+ target_name = dbt.config.get("target_name")
+ specific_var = dbt.config.get("specific_var")
+ specific_env_var = dbt.config.get("specific_env_var")
+
+ orders_df = dbt.ref("fct_orders")
+
+ # limit data in dev
+ if target_name == "dev":
+ orders_df = orders_df.limit(500)
+```
+
+
+
+### Materializations
+
+Python models support two materializations:
+- `table`
+- `incremental`
+
+Incremental Python models support all the same [incremental strategies](/docs/build/incremental-models#about-incremental_strategy) as their SQL counterparts. The specific strategies supported depend on your adapter.
+
+Python models can't be materialized as `view` or `ephemeral`. Python isn't supported for non-model resource types (like tests and snapshots).
+
+For incremental models, like SQL models, you will need to filter incoming tables to only new rows of data:
+
+
+
+
+
+**Note:** Incremental models are supported on BigQuery/Dataproc for the `merge` incremental strategy. The `insert_overwrite` strategy is not yet supported.
+
+## Python-specific functionality
+
+### Defining functions
+
+In addition to defining a `model` function, the Python model can import other functions or define its own. Here's an example, on Snowpark, defining a custom `add_one` function:
+
+
+
+```python
+def add_one(x):
+ return x + 1
+
+def model(dbt, session):
+ dbt.config(materialized="table")
+ temps_df = dbt.ref("temperatures")
+
+ # warm things up just a little
+ df = temps_df.withColumn("degree_plus_one", add_one(temps_df["degree"]))
+ return df
+```
+
+
+
+At present, Python functions defined in one dbt model can't be imported and reused in other models. See the ["Code reuse"](#code-reuse) section for the potential patterns we're considering.
+
+### Using PyPI packages
+
+You can also define functions that depend on third-party packages, so long as those packages are installed and available to the Python runtime on your data platform. See notes on "Installing Packages" for [specific data warehouses](#specific-data-warehouses).
+
+In this example, we use the `holidays` package to determine if a given date is a holiday in France. For simplicity and consistency across platforms, the code below uses the pandas API. The exact syntax, and the need to refactor for multi-node processing, still varies.
+
+
+
+
+
+#### Configuring packages
+
+We encourage you to explicitly configure required packages and versions so dbt can track them in project metadata. This configuration is required for the implementation on some platforms. If you need specific versions of packages, specify them.
+
+
+
+```python
+def model(dbt, session):
+ dbt.config(
+ packages = ["numpy==1.23.1", "scikit-learn"]
+ )
+```
+
+
+
+
+
+```yml
+version: 2
+
+models:
+ - name: my_python_model
+ config:
+ packages:
+ - "numpy==1.23.1"
+ - scikit-learn
+```
+
+
+
+#### UDFs
+
+You can use the `@udf` decorator or `udf` function to define an "anonymous" function and call it within your `model` function's DataFrame transformation. This is a typical pattern for applying more complex functions as DataFrame operations, especially if those functions require inputs from third-party packages.
+- [Snowpark Python: Creating UDFs](https://docs.snowflake.com/en/developer-guide/snowpark/python/creating-udfs.html)
+- [PySpark functions: udf](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.udf.html)
+
+
+
+
+
+#### Code reuse
+
+Currently, you cannot import or reuse Python functions defined in one dbt model, in other models. This is something we'd like dbt to support. There are two patterns we're considering:
+1. Creating and registering **"named" UDFs**. This process is different across data platforms and has some performance limitations. (Snowpark does support ["vectorized" UDFs](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-batch.html): pandas-like functions that you can execute in parallel.)
+2. Using **private Python packages**. In addition to importing reusable functions from public PyPI packages, many data platforms support uploading custom Python assets and registering them as packages. The upload process looks different across platforms, but your code’s actual `import` looks the same.
+
+:::note ❓ Our questions
+
+- Should dbt have a role in abstracting over UDFs? Should dbt support a new type of DAG node, `function`? Would the primary use case be code reuse across Python models or defining Python-language functions that can be called from SQL models?
+- How can dbt help users when uploading or initializing private Python assets? Is this a new form of `dbt deps`?
+- How can dbt support users who want to test custom functions? If defined as UDFs: "unit testing" in the database? If "pure" functions in packages: encourage adoption of `pytest`?
+
+💬 Discussion: ["Python models: package, artifact/object storage, and UDF management in dbt"](https://github.com/dbt-labs/dbt-core/discussions/5741)
+:::
+
+### DataFrame API and syntax
+
+Over the past decade, most people writing data transformations in Python have adopted DataFrame as their common abstraction. dbt follows this convention by returning `ref()` and `source()` as DataFrames, and it expects all Python models to return a DataFrame.
+
+A DataFrame is a two-dimensional data structure (rows and columns). It supports convenient methods for transforming that data, creating new columns from calculations performed on existing columns. It also offers convenient ways for previewing data while developing locally or in a notebook.
+
+That's about where the agreement ends. There are numerous frameworks with their own syntaxes and APIs for DataFrames. The [pandas](https://pandas.pydata.org/docs/) library offered one of the original DataFrame APIs, and its syntax is the most common to learn for new data professionals. Most newer DataFrame APIs are compatible with pandas-style syntax, though few can offer perfect interoperability. This is true for Snowpark and PySpark, which have their own DataFrame APIs.
+
+When developing a Python model, you will find yourself asking these questions:
+
+**Why pandas?** It's the most common API for DataFrames. It makes it easy to explore sampled data and develop transformations locally. You can “promote” your code as-is into dbt models and run it in production for small datasets.
+
+**Why _not_ pandas?** Performance. pandas runs "single-node" transformations, which cannot benefit from the parallelism and distributed computing offered by modern data warehouses. This quickly becomes a problem as you operate on larger datasets. Some data platforms support optimizations for code written using pandas' DataFrame API, preventing the need for major refactors. For example, ["pandas on PySpark"](https://spark.apache.org/docs/latest/api/python/getting_started/quickstart_ps.html) offers support for 95% of pandas functionality, using the same API while still leveraging parallel processing.
+
+:::note ❓ Our questions
+- When developing a new dbt Python model, should we recommend pandas-style syntax for rapid iteration and then refactor?
+- Which open source libraries provide compelling abstractions across different data engines and vendor-specific APIs?
+- Should dbt attempt to play a longer-term role in standardizing across them?
+
+💬 Discussion: ["Python models: the pandas problem (and a possible solution)"](https://github.com/dbt-labs/dbt-core/discussions/5738)
+:::
+
+### Limitations
+
+Python models have capabilities that SQL models do not. They also have some drawbacks compared to SQL models:
+
+- **Time and cost.** Python models are slower to run than SQL models, and the cloud resources that run them can be more expensive. Running Python requires more general-purpose compute. That compute might sometimes live on a separate service or architecture from your SQL models. **However:** We believe that deploying Python models via dbt—with unified lineage, testing, and documentation—is, from a human standpoint, **dramatically** faster and cheaper. By comparison, spinning up separate infrastructure to orchestrate Python transformations in production and different tooling to integrate with dbt is much more time-consuming and expensive.
+- **Syntax differences** are even more pronounced. Over the years, dbt has done a lot, via dispatch patterns and packages such as `dbt_utils`, to abstract over differences in SQL dialects across popular data warehouses. Python offers a **much** wider field of play. If there are five ways to do something in SQL, there are 500 ways to write it in Python, all with varying performance and adherence to standards. Those options can be overwhelming. As the maintainers of dbt, we will be learning from state-of-the-art projects tackling this problem and sharing guidance as we develop it.
+- **These capabilities are very new.** As data warehouses develop new features, we expect them to offer cheaper, faster, and more intuitive mechanisms for deploying Python transformations. **We reserve the right to change the underlying implementation for executing Python models in future releases.** Our commitment to you is around the code in your model `.py` files, following the documented capabilities and guidance we're providing here.
+
+As a general rule, if there's a transformation you could write equally well in SQL or Python, we believe that well-written SQL is preferable: it's more accessible to a greater number of colleagues, and it's easier to write code that's performant at scale. If there's a transformation you _can't_ write in SQL, or where ten lines of elegant and well-annotated Python could save you 1000 lines of hard-to-read Jinja-SQL, Python is the way to go.
+
+## Specific data platforms
+
+In their initial launch, Python models are supported on three of the most popular data platforms: Snowflake, Databricks, and BigQuery/GCP (via Dataproc). Both Databricks and GCP's Dataproc use PySpark as the processing framework. Snowflake uses its own framework, Snowpark, which has many similarities to PySpark.
+
+
+
+
+
+
diff --git a/website/docs/docs/building-a-dbt-project/metrics.md b/website/docs/docs/building-a-dbt-project/metrics.md
deleted file mode 100644
index 3ca76eb0c63..00000000000
--- a/website/docs/docs/building-a-dbt-project/metrics.md
+++ /dev/null
@@ -1,139 +0,0 @@
----
-title: "Metrics"
-id: "metrics"
-description: "When you define metrics in dbt projects, you encode crucial business logic in tested, version-controlled code. The dbt metrics layer helps you standardize metrics within your organization."
-keywords:
- - dbt metrics layer
----
-
-
-
-* **v1.0.0**: Metrics are new and experimental
-
-
-
-
-:::info Metrics are new
-v1.0.0 includes an initial version of metrics, following a [vibrant community discussion](https://github.com/dbt-labs/dbt-core/issues/4071). Try them out, and let us know what you think!
-:::
-
-:::caution Metrics are experimental
-v1.0 includes metrics, but they should be considered an _unstable_ API because they are experimental and subject to change. We reserve the right to make breaking changes to the metrics schema in future **minor** versions, but will aim for backwards compatibility when possible.
-:::
-
-## About Metrics
-
-A metric is a timeseries aggregation over a
-
-```yaml
-# models/marts/product/schema.yml
-
-version: 2
-
-models:
- - name: dim_customers
- ...
-
-metrics:
- - name: new_customers
- label: New Customers
- model: ref('dim_customers')
- description: "The number of paid customers using the product"
-
- type: count
- sql: user_id # superfluous here, but shown as an example
-
- timestamp: signup_date
- time_grains: [day, week, month]
-
- dimensions:
- - plan
- - country
-
- filters:
- - field: is_paying
- operator: 'is'
- value: 'true'
- - field: lifetime_value
- operator: '>='
- value: '100'
- - field: company_name
- operator: '!='
- value: "'Acme, Inc'"
- - field: signup_date
- operator: '>='
- value: "'2020-01-01'"
-
-
- meta: {team: Finance}
-```
-
-
-
-### Available properties
-
-| Field | Description | Example | Required? |
-|-------------|-------------------------------------------------------------|---------------------------------|-----------|
-| name | A unique identifier for the metric | new_customers | yes |
-| model | The dbt model that powers this metric | dim_customers | yes |
-| label | A short for name / label for the metric | New Customers | no |
-| description | Long form, human-readable description for the metric | The number of customers who.... | no |
-| type | The type of calculation to perform when evaluating a metric | count_distinct | yes |
-| sql | The expression to aggregate/calculate over | user_id | yes |
-| timestamp | The time-based component of the metric | signup_date | yes |
-| time_grains | One or more "grains" at which the metric can be evaluated | [day, week, month] | yes |
-| dimensions | A list of dimensions to group or filter the metric by | [plan, country] | no |
-| filters | A list of filters to apply before calculating the metric | See below | no |
-| meta | Arbitrary key/value store | {team: Finance} | no |
-
-### Filters
-Filters should be defined as a list of dictionaries that define predicates for the metric. Filters are combined using AND clauses. For more control, users can (and should) include the complex logic in the model powering the metric.
-
-All three properties (`field`, `operator`, `value`) are required for each defined filter.
-
-Note that `value` must be defined as a string in YAML, because it will be compiled into queries as part of a string. If your filter's value needs to be surrounded in quotes inside the query (e.g. text or dates), use `"'nested'"` quotes:
-
-```yml
- filters:
- - field: is_paying
- operator: 'is'
- value: 'true'
- - field: lifetime_value
- operator: '>='
- value: '100'
- - field: company_name
- operator: '!='
- value: "'Acme, Inc'"
- - field: signup_date
- operator: '>='
- value: "'2020-01-01'"
-```
-
-## Ongoing discussions
-
-- Should metrics be defined on top of more strongly typed **attributes**, rather than columns? [dbt-core#4090](https://github.com/dbt-labs/dbt-core/issues/4090)
-- Should metrics include support for joins? How should dbt know about foreign-key relationships between models? [dbt-core#4125](https://github.com/dbt-labs/dbt-core/issues/4125)
-- Should metrics inherit configurations from the models on which they are defined? Should it be possible to define metrics directly on models/columns, like tests?
-
-These are just a start! We welcome you to check out open issues on GitHub, and join the conversation.
-
diff --git a/website/docs/docs/building-a-dbt-project/projects.md b/website/docs/docs/building-a-dbt-project/projects.md
deleted file mode 100644
index 8ae94adf047..00000000000
--- a/website/docs/docs/building-a-dbt-project/projects.md
+++ /dev/null
@@ -1,94 +0,0 @@
----
-title: "Projects"
-id: "projects"
----
-
-## Related reference docs
-* [`dbt_project.yml` configurations](reference/dbt_project.yml.md)
-* The [`dbt init` command](init)
-
-## Getting started
-A dbt project is a directory of `.sql` and `.yml` files, which dbt uses to transform your data. At a minimum, a dbt project must contain:
-* A project file: A `dbt_project.yml` file tells dbt that a particular directory is a dbt project, and also contains configurations for your project.
-* [Models](building-models): A model is a single `.sql` file. Each model contains a single select statement that either transforms raw data into a dataset that is ready for analytics, or, more often, is an intermediate step in such a transformation.
-
-A project may also contain a number of other resources, such as [snapshots](snapshots), [seeds](seeds), [tests](building-a-dbt-project/tests), [macros](jinja-macros#macros), [documentation](documentation), and [sources](using-sources).
-
-## Creating a dbt project
-
-:::info Creating your first dbt project
-
-If you're new to dbt, we recommend that you check out our [Getting Started Tutorial](tutorial/getting-started.md) to build your first dbt project.
-
-:::
-
-If you don't yet have a dbt project, follow these instructions to create one. The dbt [starter project](https://github.com/dbt-labs/dbt-starter-project) contains default configurations as well as helpful notes.
-
-
-
-
-
-To create a new dbt project when developing in dbt Cloud:
-
-1. Create a dbt Cloud account [here](https://cloud.getdbt.com/signup/). If your organization already has a dbt Cloud account, ask an admin to add you as a Developer.
-2. If you created a new account, a new project should automatically be created. If you were added to an existing account:
- * Click the hamburger menu, then `Account Settings`, then `Projects`.
- * Name your project, and click `Save`. There's no need to fill in the other details.
- * Click the hamburger menu, and then `Home`.
- * Switch the project in the header bar to your new "dbt Tutorial" project.
-3. Complete the project setup flow:
- * Connect to your data warehouse
- * Add a repository — either choose a managed repository, or connect to an existing, but bare, repository.
-
-
-
-
-To create a new dbt project, run:
-
-```bash
-$ dbt init [project_name]
-```
-
-This will create a new directory in your current path (i.e. at `./[project-name]`.
-
-
-
-
-
-### FAQs
-
-
-
-If your organization already has a dbt Cloud account, and you wish to develop your project using dbt Cloud, ask an admin to add you as a Developer.
-
-
-To work on an existing project:
-1. Ensure that the project is checked into a hosted git repository (for example, on GitHub, GitLab or BitBucket) that you have access to.
-2. Clone the repository to your computer.
-
-
diff --git a/website/docs/docs/collaborate/cloud-build-and-view-your-docs.md b/website/docs/docs/collaborate/cloud-build-and-view-your-docs.md
new file mode 100644
index 00000000000..d8fb539feca
--- /dev/null
+++ b/website/docs/docs/collaborate/cloud-build-and-view-your-docs.md
@@ -0,0 +1,55 @@
+---
+title: "Build and view your docs with dbt Cloud"
+id: "build-and-view-your-docs"
+description: "Automatically generate project documentation as you run jobs."
+---
+
+dbt enables you to generate documentation for your project and data warehouse, and renders the documentation in a website. For more information, see [Documentation](/docs/collaborate/documentation).
+
+## Set up a documentation job
+
+You can set up documentation for a job in dbt Cloud when you edit your job settings or create a new job. You need to configure the job to generate docs when it runs, then link that job to your project.
+
+To set up a job to generate docs:
+
+1. In the top left, click **Deploy** and select **Jobs**.
+2. Create a new job or select an existing job and click **Settings**.
+3. Under "Execution Settings," select **Generate docs on run**.
+
@@ -68,12 +66,12 @@ First, run `dbt docs generate` — this command tells dbt to compile relevant in
Then, run `dbt docs serve` to use these `.json` files to populate a local website.
## FAQs
-lineage for our model. This lineage is filterable using the `--select` and `--exclude` flags, which are consistent with the semantics of [model selection syntax](node-selection/syntax). Further, you can right-click to interact with the DAG, jump to documentation, or share links to your graph visualization with your coworkers.
views ) that your end users query (often through a BI tool).
+Data warehouses can also be designed to have separate environments – the _production_ environment refers to the relations (i.e. schemas, tables, and views ) that your end users query (often through a BI tool).
## How do I maintain different environments with dbt?
dbt makes it easy to maintain separate production and development environments through the use of targets within a profile. A typical profile when using dbt locally (i.e. running from your command line) will have a target named `dev`, and have this set as the default. This means that while making changes, your objects will be built in your _development_ target, without affecting production queries made by your end users. Once you are confident in your changes, you can deploy the code to _production_, by running your dbt project with a _prod_ target.
:::info Running dbt in production
-You can learn more about different ways to run dbt in production in [this article](running-dbt-in-production)
+You can learn more about different ways to run dbt in production in [this article](/docs/deploy/deployments).
:::
Targets offer the flexibility to decide how to implement your separate environments – whether you want to use separate schemas, databases, or entirely different clusters altogether! We recommend using _different schemas within one data warehouse_ to separate your environments. This is the easiest to set up, and is the most cost effective solution in a modern cloud-based data stack.
In practice, this means that most of the details in a target will be consistent across all targets, except for the `schema` and user credentials. If you have multiple dbt users writing code, it often makes sense for _each user_ to have their own _development_ environment. A pattern we've found useful is to set your dev target schema to be `dbt_`. User credentials should also differ across targets so that each dbt user is using their own data warehouse user.
+
+## Related docs
+- [About dbt Core versions](/docs/dbt-versions/core)
+- [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-core-in-cloud)
diff --git a/website/docs/docs/collaborate/git-version-control.md b/website/docs/docs/collaborate/git-version-control.md
new file mode 100644
index 00000000000..7006b8836f2
--- /dev/null
+++ b/website/docs/docs/collaborate/git-version-control.md
@@ -0,0 +1,24 @@
+---
+title: "About git"
+id: git-version-control
+description: "Git overview"
+sidebar_label: "About git"
+---
+
+A [version control](https://en.wikipedia.org/wiki/Version_control) system allows you and your teammates to work collaboratively, safely, and simultaneously on a single project. Version control helps you track all the code changes made in your dbt project.
+
+In a distributed version control system, every developer has a full copy of the project and project history. Git is one of the most popular distributed version control systems and is commonly used for both open source and commercial software development, with great benefits for individuals, teams and businesses.
+
+
+
+
+Git allows developers see the entire timeline of their changes, decisions, and progression of any project in one place. From the moment they access the history of a project, the developer has all the context they need to understand it and start contributing.
+
+When you develop in the command line interface (CLI) or Cloud integrated development environment (IDE), you can leverage Git directly to version control your code. To use version control, make sure you are connected to a Git repository in the CLI or Cloud IDE.
+
+
+## Related docs
+- [Version control basics](/docs/collaborate/git/version-control-basics)
+- [Resolve merge conflicts](/docs/collaborate/git/resolve-merge-conflicts)
+- [Connect to GitHub](/docs/collaborate/git/connect-github)
+- [Connect to GitLab](/docs/collaborate/git/connect-gitlab)
diff --git a/website/docs/docs/collaborate/git/authenticate-azure.md b/website/docs/docs/collaborate/git/authenticate-azure.md
new file mode 100644
index 00000000000..abac4fd1b59
--- /dev/null
+++ b/website/docs/docs/collaborate/git/authenticate-azure.md
@@ -0,0 +1,24 @@
+---
+title: "Authenticate with Azure DevOps"
+id: "authenticate-azure"
+description: "dbt Cloud developers need to authenticate with Azure DevOps."
+sidebar_label: "Authenticate with Azure DevOps"
+---
+
+
+If you use the dbt Cloud IDE to collaborate on your team's Azure DevOps dbt repo, you need to [link your dbt Cloud profile to Azure DevOps](#link-your-dbt-cloud-profile-to-azure-devops), which provides an extra layer of authentication.
+
+## Link your dbt Cloud profile to Azure DevOps
+
+Connect your dbt Cloud profile to Azure DevOps using OAuth:
+
+1. Click the gear icon at the top right and select **Profile settings**.
+2. Click **Linked Accounts**.
+3. Next to Azure DevOps, click **Link**.
+
Note: This option will not display if your repo isn't completely empty (i.e. includes a README file).
Once you click **Initialize your project**, click **Commit** to finish setting up your project. | +| Open pull request | This allows you to open a pull request in Git for peers to review changes before merging into the base branch. | +| Pull changes from master/main | This option is available if you are on any local branch that is behind the remote version of the base branch or the remote version of the branch that you're currently on. | +| Pull from remote | This option is available if you’re on the local base branch and changes have recently been pushed to the remote version of the branch. Pulling in changes from the remote repo allows you to pull in the most recent version of the base branch. | +| Reclone Your Repository | Reclone your repository directly from the Cloud IDE. You can reset your repository back to a fresh clone from your remote. To do this, click on the bottom right-hand side green **Ready** text, then click **Reclone Repo**. | +| Refresh git state | This enables you to pull new branches from a different remote branch to your local branch with just one command. | + + +## Merge conflicts + +Merge conflicts often occur when multiple users are concurrently making edits to the same section in the same file. This makes it difficult for Git to determine which change should be kept. + +Refer to [resolve merge conflicts](/docs/collaborate/git/resolve-merge-conflicts) to learn how to resolve merge conflicts. + +## The .gitignore file + +dbt Labs recommends that you exclude files so they're not tracked by Git and won't slow down your dbt project. + +You can do this with a special file named [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) which is automatically included in your dbt project after you initialize it in dbt Cloud. The `.gitignore` file must be placed at the root of your dbt project. diff --git a/website/docs/docs/collaborate/manage-access/about-access.md b/website/docs/docs/collaborate/manage-access/about-access.md new file mode 100644 index 00000000000..844f49e6500 --- /dev/null +++ b/website/docs/docs/collaborate/manage-access/about-access.md @@ -0,0 +1,150 @@ +--- +title: "About access" +id: "about-access" +--- + +## Overview + +dbt Cloud administrators can use dbt Cloud's permissioning model to control +user-level access in a dbt Cloud account. This access control comes in two flavors: +License-based and Role-based. + +- **License-based Access Controls:** User are configured with account-wide + license types. These licenses control the specific parts of the dbt Cloud application + that a given user can access. +- **Role-based Access Control (RBAC):** Users are assigned to _groups_ that have + specific permissions on specific projects or the entire account. A user may be + a member of multiple groups, and those groups may have permissions on multiple + projects. + +## License-based access control + +Each user on an account is assigned a license type when the user is first +invited to a given account. This license type may change over time, but a +user can only have one type of license at any given time. + +A user's license type controls the features in dbt Cloud that the user is able +to access. dbt Cloud's two license types are: + - **Read Only** + - **Developer** + +For more information on these license types, see [Seats & Users](cloud-seats-and-users). +At a high level, Developers may be granted _any_ permissions, whereas Read Only +users will have read-only permissions applied to all dbt Cloud resources +regardless of the role-based permissions that the user is assigned. + +## Role-based access control + +:::info dbt Cloud Enterprise + +Role-based access control is a feature of the dbt Cloud Enterprise plan + +::: + +Role-based access control allows forfine-grained permissioning in the dbt Cloud
+application. With role-based access control, users can be assigned varying
+permissions to different projects within a dbt Cloud account. For teams on the
+Enterprise tier, role-based permissions can be generated dynamically from
+configurations in an [Identity Provider](sso-overview).
+
+Role-based permissions are applied to _groups_ and pertain to _projects_. The
+assignable permissions themselves are granted via _permission sets_.
+
+
+### Groups
+
+A group is a collection of users. Users may belong to multiple groups. Members
+of a group inherit any permissions applied to the group itself.
+
+Users can be added to a dbt Cloud group based on their group memberships in the
+configured [Identity Provider](sso-overview) for the account. In this way, dbt
+Cloud administrators can manage access to dbt Cloud resources via identity
+management software like Azure AD, Okta, or GSuite. See _SSO Mappings_ below for
+more information.
+
+You can view the groups in your account or create new groups from the **Team > Groups**
+page in your Account Settings.
+
+
+
+- New in version 1.1.23 (March, 2021)
+
+
+
+dbt Cloud administrators can manually assign users to groups independently of
+IdP attributes. If a dbt Cloud group is configured _without_ any
+SSO Mappings, then the group will be _unmanaged_ and dbt Cloud will not adjust
+group membership automatically when users log into dbt Cloud via an identity
+provider. This behavior may be desirable for teams that have connected an identity
+provider, but have not yet configured SSO Mappings between dbt Cloud and the
+IdP.
+
+If an SSO Mapping is added to an _unmanaged_ group, then it will become
+_managed_, and dbt Cloud may add or remove users to the group automatically at
+sign-in time based on the user's IdP-provided group membership information.
+
+
+## FAQs
+- **When are IdP group memberships updated for SSO Mapped groups?** Group memberships
+ are updated every time a user logs into dbt Cloud via a supported SSO provider. If
+ you've changed group memberships in your identity provider or dbt Cloud, ask your
+ users to log back into dbt Cloud for these group memberships to be synchronized.
+
+- **Can I set up SSO without RBAC?** Yes, see the documentation on
+ [Manual Assignment](#manual-assignment) above for more information on using
+ SSO without RBAC.
+
+- **Can I configure a user's License Type based on IdP Attributes?** Yes, see
+ the docs on [managing license types](/cloud-seats-and-users#managing-license-types)
+ for more information.
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-enterprise/audit-log.md b/website/docs/docs/collaborate/manage-access/audit-log.md
similarity index 87%
rename from website/docs/docs/dbt-cloud/dbt-cloud-enterprise/audit-log.md
rename to website/docs/docs/collaborate/manage-access/audit-log.md
index 698b4f90071..78d59d9a0a2 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-enterprise/audit-log.md
+++ b/website/docs/docs/collaborate/manage-access/audit-log.md
@@ -1,19 +1,22 @@
---
-title: "Using the audit log for dbt Cloud Enterprise"
+title: "The audit log for dbt Cloud Enterprise"
id: audit-log
description: "You can troubleshoot possible issues and provide security audits by reviewing event activity in your organization."
-sidebar_label: "Using the audit log"
+sidebar_label: "Audit log"
---
-To review actions performed by people in your organization, dbt provides logs of audited user and system events. The dbt Cloud audit log lists events triggered in your organization within the last 90 days.
-
-Use the audit log to quickly review the actions performed by members of your organization. The audit log includes details such as who performed the action, what the action was, and when it was performed. You can use these details to troubleshoot access issues, perform security audits, or analyze specific events.
+To review actions performed by people in your organization, dbt provides logs of audited user and system events. You can use the audit log to quickly review the actions performed by members of your organization. The audit log includes details such as who performed the action, what the action was, and when it was performed. You can use these details to troubleshoot access issues, perform security audits, or analyze specific events.
You must be an **Account Admin** to access the audit log and this feature is only available on Enterprise plans.
+The dbt Cloud audit log stores all the events that occurred in your organization:
+
+- For events within 90 days, the dbt Cloud audit log has a selectable date range that lists events triggered.
+- For events beyond 90 days, **Account Admins** can [export all events](#exporting-logs) by using **Export All**.
+
## Accessing the audit log
-To access audit log, click **Account Settings** and **Audit Log**.
+To access audit log, click the gear icon in the top right, then click **Audit Log**.
Details for the check strategy
-Snapshot query results at `2019-01-01 11:00`: +Snapshot query results at `2019-01-01 11:00`
| id | status | | -- | ------- | @@ -406,9 +409,9 @@ Snapshot results: ## FAQs -
+
+
+
+```python
+import snowflake.snowpark.functions as F
+
+def model(dbt, session):
+ dbt.config(materialized = "incremental")
+ df = dbt.ref("upstream_table")
+
+ if dbt.is_incremental:
+
+ # only new rows compared to max in current table
+ max_from_this = f"select max(updated_at) from {dbt.this}"
+ df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0])
+
+ # or only rows from the past 3 days
+ df = df.filter(df.updated_at >= F.dateadd("day", F.lit(-3), F.current_timestamp()))
+
+ ...
+
+ return df
+```
+
+
+
+
+
+
+
+
+
+```python
+import pyspark.sql.functions as F
+
+def model(dbt, session):
+ dbt.config(materialized = "incremental")
+ df = dbt.ref("upstream_table")
+
+ if dbt.is_incremental:
+
+ # only new rows compared to max in current table
+ max_from_this = f"select max(updated_at) from {dbt.this}"
+ df = df.filter(df.updated_at >= session.sql(max_from_this).collect()[0][0])
+
+ # or only rows from the past 3 days
+ df = df.filter(df.updated_at >= F.date_add(F.current_timestamp(), F.lit(-3)))
+
+ ...
+
+ return df
+```
+
+
+
+
+
+
+
+
+
+```python
+import holidays
+
+def is_holiday(date_col):
+ # Chez Jaffle
+ french_holidays = holidays.France()
+ is_holiday = (date_col in french_holidays)
+ return is_holiday
+
+def model(dbt, session):
+ dbt.config(
+ materialized = "table",
+ packages = ["holidays"]
+ )
+
+ orders_df = dbt.ref("stg_orders")
+
+ df = orders_df.to_pandas()
+
+ # apply our function
+ # (columns need to be in uppercase on Snowpark)
+ df["IS_HOLIDAY"] = df["ORDER_DATE"].apply(is_holiday)
+
+ # return final dataset (Pandas DataFrame)
+ return df
+```
+
+
+
+
+
+
+
+
+
+```python
+import holidays
+
+def is_holiday(date_col):
+ # Chez Jaffle
+ french_holidays = holidays.France()
+ is_holiday = (date_col in french_holidays)
+ return is_holiday
+
+def model(dbt, session):
+ dbt.config(
+ materialized = "table",
+ packages = ["holidays"]
+ )
+
+ orders_df = dbt.ref("stg_orders")
+
+ df = orders_df.to_pandas_on_spark() # Spark 3.2+
+ # df = orders_df.toPandas() in earlier versions
+
+ # apply our function
+ df["is_holiday"] = df["order_date"].apply(is_holiday)
+
+ # convert back to PySpark
+ df = df.to_spark() # Spark 3.2+
+ # df = session.createDataFrame(df) in earlier versions
+
+ # return final dataset (PySpark DataFrame)
+ return df
+```
+
+
+
+
+
+
+
+
+
+```python
+import snowflake.snowpark.types as T
+import snowflake.snowpark.functions as F
+import numpy
+
+def register_udf_add_random():
+ add_random = F.udf(
+ # use 'lambda' syntax, for simple functional behavior
+ lambda x: x + numpy.random.normal(),
+ return_type=T.FloatType(),
+ input_types=[T.FloatType()]
+ )
+ return add_random
+
+def model(dbt, session):
+
+ dbt.config(
+ materialized = "table",
+ packages = ["numpy"]
+ )
+
+ temps_df = dbt.ref("temperatures")
+
+ add_random = register_udf_add_random()
+
+ # warm things up, who knows by how much
+ df = temps_df.withColumn("degree_plus_random", add_random("degree"))
+ return df
+```
+
+
+
+**Note:** Due to a Snowpark limitation, it is not currently possible to register complex named UDFs within stored procedures, and therefore dbt Python models. We are looking to add native support for Python UDFs as a project/DAG resource type in a future release. For the time being, if you want to create a "vectorized" Python UDF via the Batch API, we recommend either:
+- Writing [`create function`](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-batch.html) inside a SQL macro, to run as a hook or run-operation
+- [Registering from a staged file](https://docs.snowflake.com/ko/developer-guide/snowpark/reference/python/_autosummary/snowflake.snowpark.udf.html#snowflake.snowpark.udf.UDFRegistration.register_from_file) within your Python model code
+
+
+
+
+
+
+
+```python
+from pyspark.sql.types as T
+import pyspark.sql.functions as F
+import numpy
+
+# use a 'decorator' for more readable code
+@F.udf(returnType=T.DoubleType())
+def add_random(x):
+ random_number = numpy.random.normal()
+ return x + random_number
+
+def model(dbt, session):
+ dbt.config(
+ materialized = "table",
+ packages = ["numpy"]
+ )
+
+ temps_df = dbt.ref("temperatures")
+
+ # warm things up, who knows by how much
+ df = temps_df.withColumn("degree_plus_random", add_random("degree"))
+ return df
+```
+
+
+
+
+
+
+
+**Additional setup:** You will need to [acknowledge and accept Snowflake Third Party Terms](https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-packages.html#getting-started) to use Anaconda packages.
+
+**Installing packages:** Snowpark supports several popular packages via Anaconda. The complete list is at https://repo.anaconda.com/pkgs/snowflake/. Packages are installed at the time your model is being run. Different models can have different package dependencies. If you are using third-party packages, Snowflake recommends using a dedicated virtual warehouse for best performance rather than one with many concurrent users.
+
+**About "sprocs":** dbt submits Python models to run as "stored procedures," which some people call "sprocs" for short. By default, dbt will create a named sproc containing your model's compiled Python code, and then "call" it to execute. Snowpark has a Private Preview feature for "temporary" or "anonymous" stored procedures ([docs](https://docs.snowflake.com/en/LIMITEDACCESS/call-with.html)), which are faster and leave a cleaner query history. If this feature is enabled for your account, you can switch it on for your models by configuring `use_anonymous_sproc: True`. We plan to switch this on for all dbt + Snowpark Python models in a future release.
+
+
+
+```yml
+# I asked Snowflake Support to enable this Private Preview feature,
+# and now my dbt-py models run even faster!
+models:
+ use_anonymous_sproc: True
+```
+
+
+
+**Docs:** ["Developer Guide: Snowpark Python"](https://docs.snowflake.com/en/developer-guide/snowpark/python/index.html)
+
+
+
+
+
+**Submission methods:** Databricks supports a few different mechanisms to submit PySpark code, each with relative advantages. Some are better for supporting iterative development, while others are better for supporting lower-cost production deployments. The options are:
+- `all_purpose_cluster` (default): dbt will run your Python model using the cluster ID configured as `cluster` in your connection profile or for this specific model. These clusters are more expensive but also much more responsive. We recommend using an interactive all-purpose cluster for quicker iteration in development.
+ - `create_notebook: True`: dbt will upload your model's compiled PySpark code to a notebook in the namespace `/Shared/dbt_python_model/{schema}`, where `{schema}` is the configured schema for the model, and execute that notebook to run using the all-purpose cluster. The appeal of this approach is that you can easily open the notebook in the Databricks UI for debugging or fine-tuning right after running your model. Remember to copy any changes into your dbt `.py` model code before re-running.
+ - `create_notebook: False` (default): dbt will use the [Command API](https://docs.databricks.com/dev-tools/api/1.2/index.html#run-a-command), which is slightly faster.
+- `job_cluster`: dbt will upload your model's compiled PySpark code to a notebook in the namespace `/Shared/dbt_python_model/{schema}`, where `{schema}` is the configured schema for the model, and execute that notebook to run using a short-lived jobs cluster. For each Python model, Databricks will need to spin up the cluster, execute the model's PySpark transformation, and then spin down the cluster. As such, job clusters take longer before and after model execution, but they're also less expensive, so we recommend these for longer-running Python models in production. To use the `job_cluster` submission method, your model must be configured with `job_cluster_config`, which defines key-value properties for `new_cluster`, as defined in the [JobRunsSubmit API](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsRunsSubmit).
+
+You can configure each model's `submission_method` in all the standard ways you supply configuration:
+
+```python
+def model(dbt, session):
+ dbt.config(
+ submission_method="all_purpose_cluster",
+ create_notebook=True,
+ cluster_id="abcd-1234-wxyz"
+ )
+ ...
+```
+```yml
+version: 2
+models:
+ - name: my_python_model
+ config:
+ submission_method: job_cluster
+ job_cluster_config:
+ spark_version: ...
+ node_type_id: ...
+```
+```yml
+# dbt_project.yml
+models:
+ project_name:
+ subfolder:
+ # set defaults for all .py models defined in this subfolder
+ +submission_method: all_purpose_cluster
+ +create_notebook: False
+ +cluster_id: abcd-1234-wxyz
+```
+
+If not configured, `dbt-spark` will use the built-in defaults: the all-purpose cluster (based on `cluster` in your connection profile) without creating a notebook. The `dbt-databricks` adapter will default to the cluster configured in `http_path`. We encourage explicitly configuring the clusters for Python models in Databricks projects.
+
+**Installing packages:** When using all-purpose clusters, we recommend installing packages which you will be using to run your Python models.
+
+**Docs:**
+- [PySpark DataFrame syntax](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.html)
+- [Databricks: Introduction to DataFrames - Python](https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html)
+
+
+
+
+
+The `dbt-bigquery` adapter uses a service called Dataproc to submit your Python models as PySpark jobs. That Python/PySpark code will read from your tables and views in BigQuery, perform all computation in Dataproc, and write the final result back to BigQuery.
+
+**Submission methods.** Dataproc supports two submission methods: `serverless` and `cluster`. Dataproc Serverless does not require a ready cluster, which saves on hassle and cost—but it is slower to start up, and much more limited in terms of available configuration. For example, Dataproc Serverless supports only a small set of Python packages, though it does include `pandas`, `numpy`, and `scikit-learn`. (See the full list [here](https://cloud.google.com/dataproc-serverless/docs/guides/custom-containers#example_custom_container_image_build), under "The following packages are installed in the default image"). Whereas, by creating a Dataproc Cluster in advance, you can fine-tune the cluster's configuration, install any PyPI packages you want, and benefit from faster, more responsive runtimes.
+
+Use the `cluster` submission method with dedicated Dataproc clusters you or your organization manage. Use the `serverless` submission method to avoid managing a Spark cluster. The latter may be quicker for getting started, but both are valid for production.
+
+**Additional setup:**
+- Create or use an existing [Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets)
+- Enable Dataproc APIs for your project + region
+- If using the `cluster` submission method: Create or use an existing [Dataproc cluster](https://cloud.google.com/dataproc/docs/guides/create-cluster) with the [Spark BigQuery connector initialization action](https://github.com/GoogleCloudDataproc/initialization-actions/tree/master/connectors#bigquery-connectors). (Google recommends copying the action into your own Cloud Storage bucket, rather than using the example version shown in the screenshot below.)
+
+
+
+
+
+ +**Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 + +**Namespace:** ServiceHooks + +**Permission:** +```json +{ + "bit": 1, + "displayName": "View Subscriptions", + "name": "ViewSubscriptions" +} +``` + +**Uses:** To view existing Azure DevOps service hooks subscriptions + +**Token (where applicable - API only):** +- PublisherSecurity for access to all projects +- PublisherSecurity/ for per project access
+
+**UI/API:** API only
+
+
+
+ViewSubscriptions
++ +**Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 + +**Namespace:** ServiceHooks + +**Permission:** +```json +{ + "bit": 1, + "displayName": "View Subscriptions", + "name": "ViewSubscriptions" +} +``` + +**Uses:** To view existing Azure DevOps service hooks subscriptions + +**Token (where applicable - API only):** +- PublisherSecurity for access to all projects +- PublisherSecurity/
+
+ +**Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 + +**Namespace:** ServiceHooks + +**Permission:** +```json +{ + "bit": 2, + "displayName": "Edit Subscription", + "name": "EditSubscriptions" +} + +``` + +**Uses:** To add or update existing Azure DevOps service hooks subscriptions + +**Token (where applicable - API only):** +- PublisherSecurity for access to all projects +- PublisherSecurity/ for per project access
+
+**UI/API:** API only
+
+
+
+EditSubscriptions
++ +**Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 + +**Namespace:** ServiceHooks + +**Permission:** +```json +{ + "bit": 2, + "displayName": "Edit Subscription", + "name": "EditSubscriptions" +} + +``` + +**Uses:** To add or update existing Azure DevOps service hooks subscriptions + +**Token (where applicable - API only):** +- PublisherSecurity for access to all projects +- PublisherSecurity/
+
+ +**Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 + +**Namespace:** ServiceHooks + +**Permission:** +```json +{ + "bit": 4, + "displayName": "Delete Subscriptions", + "name": "DeleteSubscriptions" +} + + +``` + +**Uses:** To delete any redundant Azure DevOps service hooks subscriptions + + +**Token (where applicable - API only):** +- PublisherSecurity for access to all projects +- PublisherSecurity/ for per project access
+
+**UI/API:** API only
+
+**Additional Notes:** This permission has been deprecated in recent Azure DevOps versions. Edit Subscriptions (bit 2) has Delete permissions.
+
+
+
+
+DeleteSubscriptions
++ +**Security Namespace ID:** cb594ebe-87dd-4fc9-ac2c-6a10a4c92046 + +**Namespace:** ServiceHooks + +**Permission:** +```json +{ + "bit": 4, + "displayName": "Delete Subscriptions", + "name": "DeleteSubscriptions" +} + + +``` + +**Uses:** To delete any redundant Azure DevOps service hooks subscriptions + + +**Token (where applicable - API only):** +- PublisherSecurity for access to all projects +- PublisherSecurity/
+
+ +**Security Namespace ID:** 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 + +**Namespace:** Git Repositories + +**Permission:** +```json +{ + "bit": 16384, + "displayName": "Contribute to pull requests", + "name": "PullRequestContribute" +} + +``` + +**Uses:** To post Pull Request statuses to Azure DevOps + + +**Token (where applicable - API only):** +- repoV2 for access to all projects +- repoV2/ for per project access
+- repoV2// for per repo access
+
+
+**UI/API:** UI and API
+
+**Additional Notes:** This permission is automatically inherited if Project Reader/Contributor/Administrator is set in the UI.
+
+
+
+
+PullRequestContribute
++ +**Security Namespace ID:** 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 + +**Namespace:** Git Repositories + +**Permission:** +```json +{ + "bit": 16384, + "displayName": "Contribute to pull requests", + "name": "PullRequestContribute" +} + +``` + +**Uses:** To post Pull Request statuses to Azure DevOps + + +**Token (where applicable - API only):** +- repoV2 for access to all projects +- repoV2/
+
+ +**Security Namespace ID:** 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 + +**Namespace:** Git Repositories + +**Permission:** +```json +{ + "bit": 4, + "displayName": "Contribute", + "name": "GenericContribute" +} + + +``` + +**Uses:** To post commit statuses to Azure DevOps + + +**Token (where applicable - API only):** +- repoV2 for access to all projects +- repoV2/ for access to a single project at a time
+- repoV2// for access to a single repo at a time
+
+
+**UI/API:** UI and API
+
+**Additional Notes:** This permission is automatically inherited if Project Contributor/Administrator is set in the UI.
+
+
+
+
+You must connect your service user before setting up a dbt Cloud project, as the the service user's permissions determine which projects dbt Cloud can import.
+
+To connect the service user:
+1. An admin must first be signed into the service user's Azure DevOps account.
+2. The admin should click **Link Azure Service User** in dbt Cloud.
+3. The admin will be directed to Azure DevOps and must accept the Azure AD app's permissions.
+4. Finally, the admin will be redirected to dbt Cloud, and the service user will be connected.
+GenericContribute
++ +**Security Namespace ID:** 2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 + +**Namespace:** Git Repositories + +**Permission:** +```json +{ + "bit": 4, + "displayName": "Contribute", + "name": "GenericContribute" +} + + +``` + +**Uses:** To post commit statuses to Azure DevOps + + +**Token (where applicable - API only):** +- repoV2 for access to all projects +- repoV2/
Note: This option will not display if your repo isn't completely empty (i.e. includes a README file).
Once you click **Initialize your project**, click **Commit** to finish setting up your project. | +| Open pull request | This allows you to open a pull request in Git for peers to review changes before merging into the base branch. | +| Pull changes from master/main | This option is available if you are on any local branch that is behind the remote version of the base branch or the remote version of the branch that you're currently on. | +| Pull from remote | This option is available if you’re on the local base branch and changes have recently been pushed to the remote version of the branch. Pulling in changes from the remote repo allows you to pull in the most recent version of the base branch. | +| Reclone Your Repository | Reclone your repository directly from the Cloud IDE. You can reset your repository back to a fresh clone from your remote. To do this, click on the bottom right-hand side green **Ready** text, then click **Reclone Repo**. | +| Refresh git state | This enables you to pull new branches from a different remote branch to your local branch with just one command. | + + +## Merge conflicts + +Merge conflicts often occur when multiple users are concurrently making edits to the same section in the same file. This makes it difficult for Git to determine which change should be kept. + +Refer to [resolve merge conflicts](/docs/collaborate/git/resolve-merge-conflicts) to learn how to resolve merge conflicts. + +## The .gitignore file + +dbt Labs recommends that you exclude files so they're not tracked by Git and won't slow down your dbt project. + +You can do this with a special file named [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) which is automatically included in your dbt project after you initialize it in dbt Cloud. The `.gitignore` file must be placed at the root of your dbt project. diff --git a/website/docs/docs/collaborate/manage-access/about-access.md b/website/docs/docs/collaborate/manage-access/about-access.md new file mode 100644 index 00000000000..844f49e6500 --- /dev/null +++ b/website/docs/docs/collaborate/manage-access/about-access.md @@ -0,0 +1,150 @@ +--- +title: "About access" +id: "about-access" +--- + +## Overview + +dbt Cloud administrators can use dbt Cloud's permissioning model to control +user-level access in a dbt Cloud account. This access control comes in two flavors: +License-based and Role-based. + +- **License-based Access Controls:** User are configured with account-wide + license types. These licenses control the specific parts of the dbt Cloud application + that a given user can access. +- **Role-based Access Control (RBAC):** Users are assigned to _groups_ that have + specific permissions on specific projects or the entire account. A user may be + a member of multiple groups, and those groups may have permissions on multiple + projects. + +## License-based access control + +Each user on an account is assigned a license type when the user is first +invited to a given account. This license type may change over time, but a +user can only have one type of license at any given time. + +A user's license type controls the features in dbt Cloud that the user is able +to access. dbt Cloud's two license types are: + - **Read Only** + - **Developer** + +For more information on these license types, see [Seats & Users](cloud-seats-and-users). +At a high level, Developers may be granted _any_ permissions, whereas Read Only +users will have read-only permissions applied to all dbt Cloud resources +regardless of the role-based permissions that the user is assigned. + +## Role-based access control + +:::info dbt Cloud Enterprise + +Role-based access control is a feature of the dbt Cloud Enterprise plan + +::: + +Role-based access control allows for
@@ -166,10 +169,11 @@ You can search the audit log to find a specific event or actor, which is limited
## Exporting logs
-You can use the audit log to export historical audit results for security, compliance, and analysis purposes. You can export data for up to the last 90 days. Click the **Export CSV** button to download a CSV file of all the events that occurred in your organization over the last 90 days.
+You can use the audit log to export all historical audit results for security, compliance, and analysis purposes:
-`. Login slugs must be unique across all dbt Cloud accounts, so pick a slug that uniquely identifies your company. |
-`
:::
-*Note:* If your dbt account is configured with a single-tenant cloud or on-premise installation, the `cloud.getdbt.com` domain in the URL above will be replaced with a different value.
+*Note:* If your dbt account instance is a VPC deployment or is [based outside the US](/docs/deploy/regions), your login URL will use the domain supplied to you by your dbt Labs account team, instead of the domain `cloud.getdbt.com`.
## Setting up RBAC
Now you have completed setting up SSO with Azure AD, the next steps will be to set up
-[RBAC groups](/docs/dbt-cloud/access-control/enterprise-permissions) to complete your access control configuration.
+[RBAC groups](/docs/collaborate/manage-access/enterprise-permissions) to complete your access control configuration.
## Troubleshooting Tips
-Ensure that the domain name under which user accounts exist in Azure matches the domain supplied in the SSO configuration on the dbt side.
+Ensure that the domain name under which user accounts exist in Azure matches the domain you supplied in [Supplying credentials](#supplying-credentials) when you configured SSO.
`. Login slugs must
be unique across all dbt Cloud accounts, so pick a slug that uniquely
@@ -118,11 +111,12 @@ Settings.
dropped into the GSuite OAuth flow and prompted to log into dbt Cloud with
your work email address. If authentication is successful, you will be
redirected back to the dbt Cloud application.
-4. On the **Verify SSO Credentials** page, verify that a `groups` entry is
+4. On the **Credentials** page, verify that a `groups` entry is
present, and that it reflects the groups you are a member of in GSuite. If
- you do not see a `groups` entry in the IdP attribute list, consult the
- Troubleshooting steps below.
- `
+
+Additionally, you may configure the IdP attributes passed from your identity provider into dbt Cloud. We recommend using the following values:
+
+
+| name | name format | value | description |
+| ---- | ----------- | ----- | ----------- |
+| email | Unspecified | ${user.email} | The user's email address |
+| first_name | Unspecified | ${user.first_name} | The user's first name |
+| last_name | Unspecified | ${user.last_name} | The user's last name |
+
+dbt Cloud's [role-based access control](/docs/collaborate/manage-access/about-access#role-based-access-control) relies
+on group mappings from the IdP to assign dbt Cloud users to dbt Cloud groups. To
+use role-based access control in dbt Cloud, also configure your identity
+provider to provide group membership information in user attribute called
+`groups`:
+
+| name | name format | value | description |
+| ---- | ----------- | ----- | ----------- |
+| groups | Unspecified | `` | The groups a user belongs to in the IdP |
+
+:::info Note
+You may use a restricted group attribute statement to limit the groups set
+to dbt Cloud for each authenticated user. For example, if all of your dbt Cloud groups start
+with `DBT_CLOUD_...`, you may optionally apply a filter like `Starts With: DBT_CLOUD_`.
+Please contact support if you have any questions.
+:::
+
+### Collect integration secrets
+
+After confirming your details, the IdP should show you the following values for
+the new SAML 2.0 integration. Keep these values somewhere safe, as you will need
+them to complete setup in dbt Cloud.
+
+- Identity Provider Issuer
+- Identity Provider SSO Url
+- X.509 Certificate
+
+### Finish setup
+
+After creating the Okta application, follow the instructions in the [dbt Cloud Setup](#dbt-cloud-setup)
+section to complete the integration.
+
+## Okta integration
+You can use the instructions in this section to configure Okta as your identity provider.
+
+1. Log into your Okta account. Using the Admin dashboard, create a new app.
+
+`
+
+
+✅ Use: `your-company-name`
+❌ Avoid: `Your-Company-Name` +4. Under the **Configuration tab**, input the following values: + + - **RelayState:** ``
+ - **Audience (EntityID):** https://cloud.getdbt.com/
+ - **ACS (Consumer) URL Validator:** https://cloud.getdbt.com/complete/saml
+ - **ACS (Consumer) URL:** https://cloud.getdbt.com/complete/saml
+
+5. Next, go to the **Parameters tab**. You must have a parameter for the Email, First Name, and Last Name attributes and include all parameters in the SAML assertions. When you add the custom parameters, make sure you check the **Include in SAML assertion** checkbox.
+
+We recommend using the following values:
+
+| name | name format | value |
+| ---- | ----------- | ----- |
+| email | Unspecified | Email |
+| first_name | Unspecified | First Name |
+| last_name | Unspecified | Last Name |
+
+dbt Cloud's [role-based access control](/docs/collaborate/manage-access/about-access#role-based-access-control) relies
+on group mappings from the IdP to assign dbt Cloud users to dbt Cloud groups. To
+use role-based access control in dbt Cloud, also configure OneLogin to provide group membership information in user attribute called
+`groups`:
+
+| name | name format | value | description |
+| ---- | ----------- | ----- | ----------- |
+| groups | Unspecified | Series of groups to be used for your organization | The groups a user belongs to in the IdP |
+
+
+### Collect integration secrets
+
+6. After confirming your details, go to the **SSO tab**. OneLogin should show you the following values for
+the new integration. Keep these values somewhere safe, as you will need them to complete setup in dbt Cloud.
+
+- Issuer URL
+- SAML 2.0 Endpoint (HTTP)
+- X.509 Certificate
+
+### Finish setup
+
+7. After creating the OneLogin application, follow the instructions in the [dbt Cloud Setup](#dbt-cloud-setup)
+section to complete the integration.
+
+## dbt Cloud Setup
+
+### Providing IdP values to dbt Cloud
+
+To complete setup, follow the steps below in dbt Cloud:
+
+1. Navigate to the **Account Settings** and then click on **Single Sign On**.
+2. Click **Edit** on the upper right corner.
+3. Provide the following SSO details:
+
+ | Field | Value |
+ | ----- | ----- |
+ | Log in with | SAML 2.0 |
+ | Identity Provider SSO Url | Paste the **Identity Provider Single Sign-On URL** shown in the IdP setup instructions |
+ | Identity Provider Issuer | Paste the **Identity Provider Issuer** shown in the IdP setup instructions |
+ | X.509 Certificate | Paste the **X.509 Certificate** shown in the IdP setup instructions |
+ | Slug | Enter your desired login slug. |
+ `
+
+### Setting up RBAC
+
+After configuring an identity provider, you will be able to set up [role-based
+access control](/docs/collaborate/manage-access/enterprise-permissions) for your account.
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-enterprise/sso-overview.md b/website/docs/docs/collaborate/manage-access/sso-overview.md
similarity index 51%
rename from website/docs/docs/dbt-cloud/dbt-cloud-enterprise/sso-overview.md
rename to website/docs/docs/collaborate/manage-access/sso-overview.md
index f47323fa886..06fd77d48fe 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-enterprise/sso-overview.md
+++ b/website/docs/docs/collaborate/manage-access/sso-overview.md
@@ -6,13 +6,13 @@ id: "sso-overview"
:::info Enterprise Feature
-This guide describes a feature of the dbt Cloud Enterprise plan.
+This guide describes a feature of the dbt Cloud Enterprise plan.
If you’re interested in learning more about an Enterprise plan, contact us at sales@getdbt.com.
:::
-This overview explains how users are provisioned in dbt Cloud via Single Sign-On (SSO).
-dbt Cloud supports JIT (Just-in-Time) provisioning and IdP-initiated login. You can learn more about our supported options [here](/docs/dbt-cloud/dbt-cloud-enterprise).
+This overview explains how users are provisioned in dbt Cloud via Single Sign-On (SSO).
+dbt Cloud supports JIT (Just-in-Time) provisioning and IdP-initiated login. You can learn more about our supported options [here](https://www.getdbt.com/pricing/).
The diagram below explains the basic process by which users are provisioned in dbt Cloud upon logging in with SSO.
@@ -33,7 +33,38 @@ The diagram below explains the basic process by which users are provisioned in d
- **Attach Matching Accounts**: dbt Cloud find all of the accounts configured to match the SSO config used by this user to log in, and then create a user license record mapping the user to the account. This step will also delete any licenses that the user should not have based on the current SSO config.
- **Attach Matching Permissions (Groups)**: dbt Cloud iterates through the groups on the matching accounts, and find all that fit one of the below catergories:
- have an SSO mapping group that is assigned to the user
- - have the "Assign by Default" option checked.
+ - have the "Assign by Default" option checked.
Then, assign all of these (and only these) to the user license. This step will also remove any permissions that the user should not have based on the current SSO group mappings.
- **dbt Cloud Application**: After these steps, the user is redirected into the dbt Cloud application, and they can begin to use the application normally.
+## SSO Enforcement
+
+:::info Security Update
+
+Please read the following update if you've enabled SSO but still have non-admin users logging in with a password. The changes outlined here will be released after September 15, 2022.
+
+:::
+
+Starting September 15, 2022, we will be making these security changes to SSO to increase the security posture of your environment:
+
+* **SSO Enforcement:** If you have SSO turned on in your organization, dbt Cloud will enforce SSO-only logins for all non-admin users. If an Account Admin already has a password, they can continue logging in with a password.
+* **SSO Re-Authentication:** dbt Cloud will prompt you to re-authenticate using your SSO provider every 24 hours to ensure high security.
+
+### How should non-admin users log in?
+
+Non-admin users that currently login with a password will no longer be able to do so. They must login using the dbt Enterprise Login URL or an identity provider (IdP). For example, Okta, Azure AD, etc.
+
+### Security best practices
+
+There are a few scenarios that might require you to login with a password. We recommend these security best-practices for the two most common scenarios:
+* **Onboarding partners and contractors** - We highly recommend that you add partners and contractors to your Identity Provider. IdPs like Okta and Azure Active Directory (AAD) offer capabilities explicitly for temporary employees. We highly recommend that you reach out to your IT team to provision an SSO license for these situations. Using an IdP highly secure, reduces any breach risk, and significantly increases the security posture of your dbt Cloud environment.
+* **Identity Provider is down -** Account admins will continue to be able to log in with a password which would allow them to work with your Identity Provider to troubleshoot the problem.
+
+### Next steps for non-admin users currently logging in with passwords
+
+If you have any non-admin users logging into dbt Cloud with a password today:
+
+1. Ensure that all users have a user account in your identity provider and are assigned dbt Cloud so they won’t lose access.
+2. Alert all dbt Cloud users that they won’t be able to use a password for logging in anymore unless they are already an Admin with a password.
+3. We **DO NOT** recommend promoting any users to Admins just to preserve password-based logins because you will reduce security of your dbt Cloud environment.
+**
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/admin-cloud-api.md b/website/docs/docs/dbt-cloud-apis/admin-cloud-api.md
similarity index 77%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/admin-cloud-api.md
rename to website/docs/docs/dbt-cloud-apis/admin-cloud-api.md
index e677299eabb..499d2e904c7 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/admin-cloud-api.md
+++ b/website/docs/docs/dbt-cloud-apis/admin-cloud-api.md
@@ -11,5 +11,3 @@ The dbt Cloud Administrative API is enabled by default for _Team_ and _Enterpris
- and more
Reference documentation for the dbt Cloud Administrative v2 API can be found [here](/dbt-cloud/api-v2).
-
-Reference documentation for the under construction dbt Cloud Administrative v4 API can be found [here](/dbt-cloud/api-v4).
diff --git a/website/docs/docs/dbt-cloud-apis/apis-overview.md b/website/docs/docs/dbt-cloud-apis/apis-overview.md
new file mode 100644
index 00000000000..fc8e7c76009
--- /dev/null
+++ b/website/docs/docs/dbt-cloud-apis/apis-overview.md
@@ -0,0 +1,18 @@
+---
+title: "APIs Overview"
+id: "overview"
+---
+
+## Overview
+
+Accounts on the _Team_ and _Enterprise_ plans can query the dbt Cloud APIs.
+
+dbt Cloud provides two APIs.
+
+The [dbt Cloud Administrative API](/docs/dbt-cloud-apis/admin-cloud-api) can be used to administrate a dbt Cloud account.
+
+The [dbt Metadata API](docs/dbt-cloud-apis/metadata-api) can be used to fetch metadata related to the state and health of your dbt project.
+
+## How to Access the APIs
+
+dbt Cloud supports two types of API Tokens: [user tokens](/docs/dbt-cloud-apis/user-tokens) and [service account tokens](/docs/dbt-cloud-apis/service-tokens). Requests to the dbt Cloud APIs can be authorized using these tokens.
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-overview.md b/website/docs/docs/dbt-cloud-apis/metadata-api.md
similarity index 71%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-overview.md
rename to website/docs/docs/dbt-cloud-apis/metadata-api.md
index db62e5e0be8..169afbb225f 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-overview.md
+++ b/website/docs/docs/dbt-cloud-apis/metadata-api.md
@@ -1,12 +1,22 @@
---
-title: "Metadata API Overview"
-id: "metadata-overview"
+title: "Metadata API"
+id: "metadata-api"
---
+## About the metadata API
+
Every time that dbt Cloud runs a dbt project, it generates metadata which pertains to the accuracy, recency, configuration, and structure of the views and tables in the warehouse. dbt Cloud serves a GraphQL API which supports arbitrary queries over this metadata; the endpoint for this API is `https://metadata.cloud.getdbt.com/graphql`. This API is an incredibly rich resource for evaluating data health longitudinally or at a point in time.
+## Prerequisites
+
+The metadata API is available to accounts on the _Team_ and _Enterprise_ plans, for any version >= dbt v0.19.0. Please note that artifacts generated with a version of dbt _less than_ v0.19.0 will not be accessible via the dbt Cloud metadata API. For information on upgrading, see "[Version migration guides](https://docs.getdbt.com/guides/migration/versions)."
+
+## How to browse the API
+
We also provide a [graphical explorer](https://metadata.cloud.getdbt.com/graphiql) for this API where you can run ad-hoc queries or browse the schema. As GraphQL provides a self-describing API, the schema shown in the GraphiQL interface is an accurate representation of the graph and fields available to query on at any point in time.
-The metadata API is available to accounts on the _Team_ and _Enterprise_ plans, for any version >= dbt v0.19.0. Please note that artifacts generated with a version of dbt _less than_ v0.19.0 will not be accessible via the dbt Cloud Metadata API.
+## Retention limits for data
+
+You can use the metadata API to query data from the previous 3 months. For example, if today was April 1, you could query data back to January 1st.
*We are continuously expanding the capabilities of the metadata API and we welcome your feedback and suggestions at metadata@dbtlabs.com.*
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-querying.md b/website/docs/docs/dbt-cloud-apis/metadata-querying.md
similarity index 82%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-querying.md
rename to website/docs/docs/dbt-cloud-apis/metadata-querying.md
index e90a13a11bc..297d76e58e1 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/metadata-querying.md
+++ b/website/docs/docs/dbt-cloud-apis/metadata-querying.md
@@ -1,12 +1,12 @@
---
-title: "Querying the Metadata API"
+title: "Query the Metadata API"
id: "metadata-querying"
---
-Accounts on the _Team_ and _Enterprise_ plans can query the dbt Metadata API.
+Accounts on the _Team_ and _Multi-Tenant Enterprise_ plans can query the dbt Metadata API.
## Authorization
-Currently, authorization of requests takes place [using a service token](https://docs.getdbt.com/docs/dbt-cloud/dbt-cloud-api/service-tokens). dbt Cloud admin users can generate a Metadata Only service token that is authorized to execute a specific query against the Metadata API.
+Currently, authorization of requests takes place [using a service token](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens). dbt Cloud admin users can generate a Metadata Only service token that is authorized to execute a specific query against the Metadata API.
Once you've created a token, you can use it in the Authorization header of requests to the dbt Cloud Metadata API. Be sure to include the Token prefix in the Authorization header, or the request will fail with a `401 Unauthorized` error. Note that `Bearer` can be used in place of `Token` in the Authorization header. Both syntaxes are equivalent.
@@ -30,7 +30,7 @@ The `` body should be a JSON string in the format:
Every query will rely on a *jobID*. You can get the jobID by clicking into the relevant job in dbt Cloud and observing the URL. In this example URL, the jobID would be 917: `https://cloud.getdbt.com/#/accounts/1/projects/665/jobs/917/`
-There are several illustrative example queries in this documentation (examples of queries on the Model node, [here](/docs/dbt-cloud/dbt-cloud-api/metadata/schema/metadata-schema-model).
+There are several illustrative example queries in this documentation (examples of queries on the Model node, [here](/docs/dbt-cloud-apis/metadata-schema-model).
## GraphiQL
You can also experiment and run queries directly in the [GraphiQL interface](https://metadata.cloud.getdbt.com/graphiql), which is convenient for exploration. On the right hand side, there is a document explorer where you can see all possible nodes and fields. Below is an example of what a query looks like in GraphiQL. Note that you must authenticate via bearer auth with your token.
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-exposure.mdx b/website/docs/docs/dbt-cloud-apis/schema-metadata-exposure.mdx
similarity index 97%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-exposure.mdx
rename to website/docs/docs/dbt-cloud-apis/schema-metadata-exposure.mdx
index 95991a19304..5297cc7da53 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-exposure.mdx
+++ b/website/docs/docs/dbt-cloud-apis/schema-metadata-exposure.mdx
@@ -5,7 +5,7 @@ id: "metadata-schema-exposure"
import { ArgsTable, SchemaTable } from "./schema";
-The exposure object allows you to query information about a particular exposure. You can learn more about exposures [here](/docs/building-a-dbt-project/exposures).
+The exposure object allows you to query information about a particular exposure. You can learn more about exposures [here](/docs/build/exposures).
### Arguments
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-exposures.mdx b/website/docs/docs/dbt-cloud-apis/schema-metadata-exposures.mdx
similarity index 96%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-exposures.mdx
rename to website/docs/docs/dbt-cloud-apis/schema-metadata-exposures.mdx
index e4d9f54f53e..54ec4bbc905 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-exposures.mdx
+++ b/website/docs/docs/dbt-cloud-apis/schema-metadata-exposures.mdx
@@ -5,7 +5,7 @@ id: "metadata-schema-exposures"
import { ArgsTable, SchemaTable } from "./schema";
-The exposures object allows you to query information about all exposures in a given job. You can learn more about exposures [here](/docs/building-a-dbt-project/exposures).
+The exposures object allows you to query information about all exposures in a given job. You can learn more about exposures [here](/docs/build/exposures).
### Arguments
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-metric.mdx b/website/docs/docs/dbt-cloud-apis/schema-metadata-metric.mdx
similarity index 89%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-metric.mdx
rename to website/docs/docs/dbt-cloud-apis/schema-metadata-metric.mdx
index 1b7d371ffa3..46a1e79abba 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-metric.mdx
+++ b/website/docs/docs/dbt-cloud-apis/schema-metadata-metric.mdx
@@ -18,7 +18,7 @@ Below we show some illustrative example queries and outline the schema (all poss
### Example Queries
#### Metric information
-The example query below outputs information about a metric. Note that you can also add any field from the Model endpoint -- here we are simply selecting name. This includes schema, database, uniqueId, columns and more -- find documentation [here](https://docs.getdbt.com/docs/dbt-cloud/dbt-cloud-api/metadata/schema/metadata-schema-model).
+The example query below outputs information about a metric. Note that you can also add any field from the Model endpoint -- here we are simply selecting name. This includes schema, database, uniqueId, columns and more -- find documentation [here](/docs/dbt-cloud-apis/metadata-schema-model).
```graphql
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-metrics.mdx b/website/docs/docs/dbt-cloud-apis/schema-metadata-metrics.mdx
similarity index 83%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-metrics.mdx
rename to website/docs/docs/dbt-cloud-apis/schema-metadata-metrics.mdx
index 48a882318c2..b9077e228bd 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-metrics.mdx
+++ b/website/docs/docs/dbt-cloud-apis/schema-metadata-metrics.mdx
@@ -51,7 +51,7 @@ The example query returns information about all metrics in this job.
```
### Fields
-metrics has access to the *same fields* as the [metric node](/docs/dbt-cloud/dbt-cloud-api/metadata/schema/metadata-schema-metric). The difference is that metrics can output a list, so instead of querying for fields for one specific metric, you can query for those parameters for all metrics in a run.
+metrics has access to the *same fields* as the [metric node](/docs/dbt-cloud-apis/metadata-schema-metric). The difference is that metrics can output a list, so instead of querying for fields for one specific metric, you can query for those parameters for all metrics in a run.
When querying for `metrics`, the following fields are available:
diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-model.mdx b/website/docs/docs/dbt-cloud-apis/schema-metadata-model.mdx
similarity index 77%
rename from website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-model.mdx
rename to website/docs/docs/dbt-cloud-apis/schema-metadata-model.mdx
index 19c2f051c6d..9643cc84399 100644
--- a/website/docs/docs/dbt-cloud/dbt-cloud-api/metadata/schema/schema-metadata-model.mdx
+++ b/website/docs/docs/dbt-cloud-apis/schema-metadata-model.mdx
@@ -56,6 +56,27 @@ The example query below could be useful if we wanted to understand information a
}
```
+#### Column-level information
+
+You can use the following example query to understand more about the columns of a given model. Note that this will only work if the job has generated documentation. For example it will work with the command `dbt docs generate`.
+
+```graphql
+{
+ model(jobId: 123, uniqueId: "model.jaffle_shop.dim_user") {
+ columns{
+ name
+ index
+ type
+ comment
+ description
+ tags
+ meta
+ }
+ }
+}
+```
+
+
### Fields
When querying for a `model`, the following fields are available:
diff --git a/website/docs/docs/dbt-cloud-apis/schema-metadata-modelByEnv.mdx b/website/docs/docs/dbt-cloud-apis/schema-metadata-modelByEnv.mdx
new file mode 100644
index 00000000000..657987039f0
--- /dev/null
+++ b/website/docs/docs/dbt-cloud-apis/schema-metadata-modelByEnv.mdx
@@ -0,0 +1,44 @@
+---
+title: "Model By Environment"
+id: "metadata-schema-modelByEnv"
+---
+
+import { ArgsTable, SchemaTable } from "./schema";
+
+
-Account Admin service tokens have full `read + write` access to an account, so please use them with caution. A Team plan refers to this permission set as an "Owner role." For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#account-admin). +Account Admin service tokens have full `read + write` access to an account, so please use them with caution. A Team plan refers to this permission set as an "Owner role." For more on these permissions, see [Account Admin](/docs/collaborate/manage-access/enterprise-permissions#account-admin). **Metadata Only**
Metadata only service tokens can authorize requests to the metadata API. @@ -43,47 +43,47 @@ Metadata only service tokens can authorize requests to the metadata API. Job admin service tokens can authorize requests for viewing, editing, and creating environments, triggering runs, and viewing historical runs. **Member**
-Member service tokens can authorize requests for viewing and editing resources, triggering runs, and inviting members to the account. Tokens assigned the Member permission set will have the same permissions as a Member user. For more information about Member users, see "[Self-service permissions](/dbt-cloud/access-control/self-service-permissions)". +Member service tokens can authorize requests for viewing and editing resources, triggering runs, and inviting members to the account. Tokens assigned the Member permission set will have the same permissions as a Member user. For more information about Member users, see "[Self-service permissions](/docs/collaborate/manage-access/self-service-permissions)". **Read-only**
Read-only service tokens can authorize requests for viewing a read-only dashboard, viewing generated documentation, and viewing source freshness reports. ### Enterprise plans using service account tokens -The following permissions can be assigned to a service account token on an Enterprise plan. For more details about these permissions, see "[Enterprise permissions](/docs/dbt-cloud/access-control/enterprise-permissions)." +The following permissions can be assigned to a service account token on an Enterprise plan. For more details about these permissions, see "[Enterprise permissions](/docs/collaborate/manage-access/enterprise-permissions)." **Account Admin**
-Account Admin service tokens have full `read + write` access to an account, so please use them with caution. For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#account-admin). +Account Admin service tokens have full `read + write` access to an account, so please use them with caution. For more on these permissions, see [Account Viewer](/docs/collaborate/manage-access/enterprise-permissions#account-admin). **Metadata Only**
Metadata only service tokens can authorize requests to the metadata API. **Job Admin**
-Job Admin service tokens can authorize request for viewing, editing, and creating environments, triggering runs, and viewing historical runs. For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#job-admin). +Job Admin service tokens can authorize request for viewing, editing, and creating environments, triggering runs, and viewing historical runs. For more on these permissions, see [Account Viewer](/docs/collaborate/manage-access/enterprise-permissions#job-admin). **Account Viewer**
-Account Viewer service tokens have read only access to dbt Cloud accounts. For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#account-viewer) on the Enterprise Permissions page. +Account Viewer service tokens have read only access to dbt Cloud accounts. For more on these permissions, see [Account Viewer](/docs/collaborate/manage-access/enterprise-permissions#account-viewer) on the Enterprise Permissions page. **Admin**
-Admin service tokens have unrestricted access to projects in dbt Cloud accounts. You have the option to grant that permission all projects in the account or grant the permission only on specific projects. For more on these permissions, see [Admin Service](docs/dbt-cloud/access-control/enterprise-permissions#admin-service) on the Enterprise Permissions page. +Admin service tokens have unrestricted access to projects in dbt Cloud accounts. You have the option to grant that permission all projects in the account or grant the permission only on specific projects. For more on these permissions, see [Admin Service](/docs/collaborate/manage-access/enterprise-permissions#admin-service) on the Enterprise Permissions page. **Git Admin**
-Git admin service tokens have all the permissions listed in [Git admin](/docs/dbt-cloud/access-control/enterprise-permissions#git-admin) on the Enterprise Permissions page. +Git admin service tokens have all the permissions listed in [Git admin](/docs/collaborate/manage-access/enterprise-permissions#git-admin) on the Enterprise Permissions page. **Database Adminn**
-Database admin service tokens have all the permissions listed in [Database admin](/docs/dbt-cloud/access-control/enterprise-permissions#database-admin) on the Enterprise Permissions page. +Database admin service tokens have all the permissions listed in [Database admin](/docs/collaborate/manage-access/enterprise-permissions#database-admin) on the Enterprise Permissions page. **Team Admin**
-Team admin service tokens have all the permissions listed in [Team admin](/docs/dbt-cloud/access-control/enterprise-permissions#team-admin) on the Enterprise Permissions page. +Team admin service tokens have all the permissions listed in [Team admin](/docs/collaborate/manage-access/enterprise-permissions#team-admin) on the Enterprise Permissions page. **Job Viewer**
-Job viewer admin service tokens have all the permissions listed in [Job viewer](/docs/dbt-cloud/access-control/enterprise-permissions#job-viewer) on the Enterprise Permissions page. +Job viewer admin service tokens have all the permissions listed in [Job viewer](/docs/collaborate/manage-access/enterprise-permissions#job-viewer) on the Enterprise Permissions page. **Developer**
-Developer service tokens have all the permissions listed in [Developer](/docs/dbt-cloud/access-control/enterprise-permissions#developer) on the Enterprise Permissions page. +Developer service tokens have all the permissions listed in [Developer](/docs/collaborate/manage-access/enterprise-permissions#developer) on the Enterprise Permissions page. **Analyst**
-Analyst admin service tokens have all the permissions listed in [Analyst](/docs/dbt-cloud/access-control/enterprise-permissions#analyst) on the Enterprise Permissions page. +Analyst admin service tokens have all the permissions listed in [Analyst](/docs/collaborate/manage-access/enterprise-permissions#analyst) on the Enterprise Permissions page. **Stakeholder**
-Stakeholder service tokens have all the permissions listed in [Stakeholder](/docs/dbt-cloud/access-control/enterprise-permissions#stakeholder) on the Enterprise Permissions page. +Stakeholder service tokens have all the permissions listed in [Stakeholder](/docs/collaborate/manage-access/enterprise-permissions#stakeholder) on the Enterprise Permissions page. diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/user-tokens.md b/website/docs/docs/dbt-cloud-apis/user-tokens.md similarity index 100% rename from website/docs/docs/dbt-cloud/dbt-cloud-api/user-tokens.md rename to website/docs/docs/dbt-cloud-apis/user-tokens.md diff --git a/website/docs/docs/dbt-cloud/access-control/self-service-permissions.md b/website/docs/docs/dbt-cloud/access-control/self-service-permissions.md deleted file mode 100644 index e9a89281a28..00000000000 --- a/website/docs/docs/dbt-cloud/access-control/self-service-permissions.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -title: "Self-Service Permissions" -id: "self-service-permissions" ---- - -## Overview - -dbt Cloud supports two different permission sets to manage permissions for self-service accounts: **Member** and **Owner**. - -The permissions afforded to each role are described below: - -| Action | Member | Owner | -| ------ | ------ | ----- | -| View and edit resources | ✅ | ✅ | -| Trigger runs | ✅ | ✅ | -| Access the IDE | ✅ | ✅ | -| Invite Members to the account | ✅ | ✅ | -| Manage billing | ❌ | ✅ | -| Manage team permissions | ❌ | ✅ | -| Invite Owners to the account | ❌ | ✅ | - -## Read Only vs. Developer License Types - -Users configured with Read Only license types will experience a restricted set of permissions in dbt Cloud. If a user is associated with a _Member_ permission set and a Read Only seat license, then they will only have access to what a Read-Only seat allows. See [Seats and Users](cloud-seats-and-users) for more information on the impact of licenses on these permissions. - -## Owner and Member Groups in dbt Cloud Enterprise - -By default, new users are added to the Member and Owner groups when they onboard to a new dbt Cloud. Member and Owner groups are included with every new dbt Cloud account. They're used to provide access for Administrators to add additional users, groups and apply permission sets. This is necessary for account onboarding but can create confusion when initially setting up SSO and RBAC for dbt Cloud Enterprise accounts as described in the [Enterprise Permissions](enterprise-permissions) guide. Owner and Member groups are **account level** groups, so their permissions override any project-level permissions you wish to apply. - -After onboarding Administrative users and configuring RBAC/SSO groups, we recommend the following steps for onboarding users to a dbt Cloud Enterprise account. - -### Create Account Admins Group - -**Important:** Do this BEFORE proceeding to the next section - -1) Create an Account Admins group -2) Assign at least one user to the Account Admins - -The assigned user will manage future group, SSO mapping and user / group assignment. - -### Remove the Owner and Member groups - -Follow the steps below for each of the Owner and Member groups in turn: - -1) Click the Hamburger icon at the top left of the Cloud Console, then Account Settings >> Groups >>
-2) Click "Edit" in the upper right corner
-
-` option. You may also manually override the dbt version of a specific job to be any of the current Core releases supported by Cloud.
+The example job seen in the screenshot above belongs to the environment "Prod". It inherits the dbt version of its environment as shown by the **Inherited from ENVIRONMENT_NAME (DBT_VERSION)** selection. You may also manually override the dbt version of a specific job to be any of the current Core releases supported by Cloud by selecting another option from the dropdown.
## Supported Versions
-We have always encouraged our customers to upgrade dbt Core versions whenever a new minor version is released. We released our first major version of dbt - `dbt v1.0` - in December 2021. Alongside this release, we updated our policy on which versions of dbt Core we will support running in dbt Cloud.
+We have always encouraged our customers to upgrade dbt Core versions whenever a new minor version is released. We released our first major version of dbt - `dbt 1.0` - in December 2021. Alongside this release, we updated our policy on which versions of dbt Core we will support in dbt Cloud.
- > **By June 30, 2022, all dbt projects in Cloud must be running v1.0 or later. Starting with v1.0, any subsequent minor versions will be allowed to run in Cloud for 1 year post release.**
-We will continue to update this table so that customers know when we plan to stop running different versions of Core in Cloud.
+ > **Starting with v1.0, any subsequent minor versions will be supported in dbt Cloud for 1 year post release. At the end of the 1 year window, accounts must upgrade to a supported version of dbt or risk service disruption.**
+
+We will continue to update this table so that customers know when we plan to stop supporting different versions of Core in dbt Cloud.
`
-
-Additionally, you may configure the IdP attributes passed from your identity
-provider into dbt Cloud. We recommend using the following values:
-
-
-| name | name format | value | description |
-| ---- | ----------- | ----- | ----------- |
-| email | Unspecified | ${user.email} | The user's email address |
-| first_name | Unspecified | ${user.first_name} | The user's first name |
-| last_name | Unspecified | ${user.last_name} | The user's last name |
-
-dbt Cloud's [role-based access control](access-control-overview#role-based-access-control) relies
-on group mappings from the IdP to assign dbt Cloud users to dbt Cloud groups. To
-use role-based access control in dbt Cloud, also configure your identity
-provider to provide group membership information in user attribute called
-`groups`:
-
-
-| name | name format | value | description |
-| ---- | ----------- | ----- | ----------- |
-| groups | Unspecified | `` | The groups a user belongs to in the IdP |
-
-Note: You may use a restricted group attribute statement to limit the groups set
-to dbt Cloud for each authenticated user. For example, if all of your dbt Cloud groups start
-with `DBT_CLOUD_...`, you may optionally apply a filter like `Starts With: DBT_CLOUD_`.
-Please contact support if you have any questions.
-
-### Collect integration secrets
-
-After confirming your details, the IdP should show you the following values for
-the new SAML 2.0 integration. Keep these values somewhere safe, as you will need
-them to complete setup in dbt Cloud.
-
-- Identity Provider Issuer
-- Identity Provider SSO Url
-- X.509 Certificate
-
-### Finish setup
-
-After creating the Okta application, follow the instructions in the [dbt Cloud Setup](#dbt-cloud-setup)
-section to complete the integration.
-
-## Okta integration
-
-_Use this section if you are configuring Okta as your identity provider_.
-
-First, log into your Okta account. Using the Admin dashboard, create a new app.
-
-`
-
-`
-
-### Setting up RBAC
-After configuring an identity provider, you will be able to set up [role-based
-access control](/access-control/enterprise-permissions) for your account.
diff --git a/website/docs/docs/dbt-cloud/deployments/deployment-overview.md b/website/docs/docs/dbt-cloud/deployments/deployment-overview.md
deleted file mode 100644
index 751198b6e29..00000000000
--- a/website/docs/docs/dbt-cloud/deployments/deployment-overview.md
+++ /dev/null
@@ -1,18 +0,0 @@
----
-id: deployment-overview
-title: Overview
----
-
-:::info Deployments
-
-This section provides an overview of the deployment types available for dbt Cloud.
-If you’re interested in learning more about an Enterprise plan, contact us at sales@getdbt.com.
-
-:::
-
-In general, the dbt Cloud application deployment models fall into two categories: **Multi Tenant** and **Single Tenant**. These deployments are hosted on infrastructure managed by dbt Labs. Both models leverage AWS infrastructure as described in the [Architecture](deployment-architecture) section.
-
-For more information on these deployment models view the below pages.
-
-- [Multi Tenant](multi-tenant-deployment)
-- [Single Tenant](single-tenant-deployment)
diff --git a/website/docs/docs/dbt-cloud/january-2020-pricing-updates.md b/website/docs/docs/dbt-cloud/january-2020-pricing-updates.md
deleted file mode 100644
index a548d8b2a3d..00000000000
--- a/website/docs/docs/dbt-cloud/january-2020-pricing-updates.md
+++ /dev/null
@@ -1,40 +0,0 @@
----
-title: "January 2020 - Pricing changes"
-id: "january-2020-pricing-updates"
----
-
-On January 14th, 2020, existing self-service dbt Cloud accounts will be migrated to one of the new pricing plans shown below. dbt Cloud account owners should receive emails containing more information about how this migration impacts their account. If you have not received such an email, please contact support at support@getdbt.com.
-
-# New Plans
-
-dbt Cloud will support two self-service plans: **Developer** and **Team**. Enterprise clients are not impacted by this change.
-
-## Pricing changes
-
-Please be aware of the following changes to dbt Cloud:
- - Accounts will no longer be charged for Users designated as "read only"
- - Accounts will no longer be charged for configured run slots
- - Standard users will be automatically migrated to Developers on January 14th, 2020. The license types available in dbt Cloud will be **Read Only** and **Developer** after January 14th.
- - You may reconfigure existing Standard users on your account to be Read Only in advance of January 14.
- - Please contact support at support@getdbt.com if you have any questions about how these pricing changes affect your account.
-
-## The Developer Plan
-
-The Developer plan provides a best-in-class analytics engineering workflow for data teams of one. The Developer plan includes:
- - 1 Developer license
- - 1 Run Slot
- - [The dbt IDE](the-dbt-ide)
- - Unlimited runs per day (previously Paid-only)
- - [PR builds](cloud-enabling-continuous-integration-with-github) (previously Paid-only)
-
-Price: **Free**
-
-## The Team Plan
-
-The Team plan is built for teams collaborating on the analytics engineering workflow. The Team plan includes:
-- Everything in the Developer Plan, plus
-- 5 run slots
-- 50 read only seats
-- API Access
-
-Price: **$50/developer/month**
diff --git a/website/docs/docs/dbt-cloud/on-premises/dependencies.md b/website/docs/docs/dbt-cloud/on-premises/dependencies.md
index de07556fb06..6278612476b 100644
--- a/website/docs/docs/dbt-cloud/on-premises/dependencies.md
+++ b/website/docs/docs/dbt-cloud/on-premises/dependencies.md
@@ -33,7 +33,7 @@ During initial installation, the KOTS appliance can be directly downloaded from
- `quay.io`: Some dependencies of Replicated are hosted as public images in the Quay.io registry.
- `hub.docker.com`: Some dependencies of Replicated are hosted as public images in Docker Hub.
-Replicated maintains a list of Replicated-owned IPs for IP access restriction purposes at https://github.com/replicatedhq/ips/blob/master/ip_addresses.json.
+Replicated maintains a list of Replicated-owned IPs for IP access restriction purposes at https://github.com/replicatedhq/ips/blob/main/ip_addresses.json.
#### dbt Cloud Appliance Installation and Upgrades
@@ -61,7 +61,7 @@ To install the dbt Cloud appliance or perform updates, some external connections
```
-Replicated maintains a list of Replicated-owned IPs for IP access restriction purposes at https://github.com/replicatedhq/ips/blob/master/ip_addresses.json.
+Replicated maintains a list of Replicated-owned IPs for IP access restriction purposes at https://github.com/replicatedhq/ips/blob/main/ip_addresses.json.
#### Ongoing Access
@@ -98,6 +98,6 @@ The required inbound ports are:
For additional information related to inbound traffic view the following sections.
-- [Application Data Flows](/docs/dbt-cloud/deployments/deployment-architecture#application-data-flows)
-- [Data Warehouse Interaction](/docs/dbt-cloud/deployments/deployment-architecture#data-warehouse-interaction)
-- [Customer Managed Network Architecture](/docs/dbt-cloud/deployments/deployment-architecture#customer-managed-general-network-architecture)
+- [Application Data Flows](docs/deploy/architecture#application-data-flows)
+- [Data Warehouse Interaction](/docs/deploy/architecture#data-warehouse-interaction)
+- [Customer Managed Network Architecture](/docs/deploy/architecture)
diff --git a/website/docs/docs/dbt-cloud/on-premises/installation.md b/website/docs/docs/dbt-cloud/on-premises/installation.md
index b8fbc58b2ed..9ba856b27f4 100644
--- a/website/docs/docs/dbt-cloud/on-premises/installation.md
+++ b/website/docs/docs/dbt-cloud/on-premises/installation.md
@@ -3,9 +3,9 @@ id: installation
title: On-Premises Installation (dbt Cloud)
---
-:::note
+:::note 📌
-We longer support new on-premises deployments, and instead have moved to a [Single Tenant](single-tenant) model hosted in the cloud
+We no longer support new on-premises deployments, and instead have moved to a [Single Tenant](/docs/deploy/single-tenant) model hosted in the cloud
:::
diff --git a/website/docs/docs/dbt-cloud/on-premises/setup.md b/website/docs/docs/dbt-cloud/on-premises/setup.md
index 1e0d9bf0516..654722192ef 100644
--- a/website/docs/docs/dbt-cloud/on-premises/setup.md
+++ b/website/docs/docs/dbt-cloud/on-premises/setup.md
@@ -79,13 +79,13 @@ After logging in, you can create a new account and invite members of your team,
Accounts and Users are separate constructs that can have a many-to-many relationship. When creating a new Account, you can add either existing or new users to the account. If one user has access to multiple accounts, they will be able to switch accounts from the dbt Cloud frontend upon login.
-Each user can have a specific role on each account. For more information on each role, please see the docs on [managing permissions](access-control-overview)
+Each user can have a specific role on each account. For more information on each role, please see the docs on [managing permissions](/docs/collaborate/manage-access/about-access)
### Deploying Application Updates
A new version of dbt Cloud will appear on the Version History page in your Configuration Console anytime any of the following happen:
-- A new version of the dbt Cloud code is released. This typically happens every two weeks, and each new version will be accompanied by a [changelog](/docs/dbt-cloud/cloud-changelog).
+- A new version of the dbt Cloud code is released. This typically happens every two weeks, and each new version will be accompanied by a [changelog](/docs/dbt-versions/dbt-cloud-release-notes).
- Any configuration change is applied to your application via the Configuration Console.
- Anytime an edit is applied to your Kubernetes configs via the overlays mechanism built into kots.
@@ -124,4 +124,4 @@ kubectl patch deployment app -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
```
-After these steps, you are ready to manage your Github integration. Your users can log into dbt Cloud, and navigate to Profile > Integrations to start connecting your Github account to dbt Cloud. See [Github](cloud-installing-the-github-application) for more details on how your users can start using the integration.
+After these steps, you are ready to manage your Github integration. Your users can log into dbt Cloud, and navigate to Profile > Integrations to start connecting your GitHub account to dbt Cloud. See [GitHub](/docs/collaborate/git/connect-github) for more details on how your users can start using the integration.
diff --git a/website/docs/docs/dbt-cloud/on-premises/usage-statistics.md b/website/docs/docs/dbt-cloud/on-premises/usage-statistics.md
index ad143e48b6e..aa7759aa4b1 100644
--- a/website/docs/docs/dbt-cloud/on-premises/usage-statistics.md
+++ b/website/docs/docs/dbt-cloud/on-premises/usage-statistics.md
@@ -21,7 +21,7 @@ Usage statistics are tracked once weekly, and include the following information:
- The number of developer and read only licenses utilized in each account
- The version of dbt Cloud installed in the on-premises environment
-This information is sent as a JSON payload to usage.getdbt.com. A typical
+This information is sent as a
-
-Only supported by v1.1 or newer.
-
-
-
-
-
-Only supported by v1.1 or newer.
-
-:::caution Experimental functionality
-The `source_status` selection is experimental and subject to change. During this time, ongoing improvements may limit this feature’s availability and cause breaking changes to its functionality.
-:::
-
-When a job is selected, dbt Cloud will surface the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of fresh sources. In your job commands, you can signal to dbt to run and test only on these fresher sources and their children by including the `source_status:fresher+` argument. This requires both previous and current state to have the `sources.json` artifact be available. Or plainly said, both job states need to run `dbt source freshness`.
-
-As example:
-```bash
-# Command step order
-dbt source freshness
-dbt build --select source_status:fresher+
-```
-
-
-More example commands in [Pro-tips for workflows](/docs/guides/best-practices.md#pro-tips-for-workflows).
-
-## Troubleshooting
-
-### Reconnecting your dbt project to use a native GitHub or GitLab repo
-
-If your dbt project relies on a Git URL and SSH to connect to your GitHub or GitLab repo, you need to disconnect your repo and reconnect it with native Github or GitLab auth in order to enable webhooks.
-
-First, make sure you have [native GitHub authentication](cloud-installing-the-github-application) or [native GitLab authentication](connecting-gitlab) set up. After you have gone through those steps, head to `Account Settings` -> `Projects` and click on the project you'd like to reconnect through native GitHub or GitLab auth. Then click on the repository link.
-
-Once you're in the repository page, you can click the `Edit` button and then the `Disconnect Repository` button at the bottom.
-fine-grained access control in their dbt projects. dbt will enforce git permissions for every developer to ensure that read / write policies in GitHub carry through to the IDE.
@@ -139,7 +139,7 @@ We shipped a far better experience for GitLab users. Be sure to check out new CI
#### New products and features
-- `Slim CI`: We’ve made Slim CI available for all our cloud customers! With Slim CI, you don't have to rebuild and test all your models; you can instruct dbt Cloud to run jobs on only modified or new resources. If you are a GitHub or GitLab user, try creating a new job that runs on pull requests and you can signal to dbt to run only on these modified resources by including the `state:modified+` argument. Read more about Slim CI [here](https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-enabling-continuous-integration-with-github#slim-ci).
+- `Slim CI`: We’ve made Slim CI available for all our cloud customers! With Slim CI, you don't have to rebuild and test all your models; you can instruct dbt Cloud to run jobs on only modified or new resources. If you are a GitHub or GitLab user, try creating a new job that runs on pull requests and you can signal to dbt to run only on these modified resources by including the `state:modified+` argument. Read more about Slim CI [here](/docs/deploy/cloud-ci-job).
- Native GitLab authentication for dbt Cloud Developer and Team Tiers: We’ve shipped native GitLab auth into GA. You can now import new GitLab repos with a couple clicks, trigger CI builds when Merge Requests are opened in GitLab, and carry GitLab permissions through to dbt Cloud IDE's git actions. Read how to set up native GitLab auth [here](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-gitlab).
@@ -178,7 +178,7 @@ A lot of improvements coming for GitLab webhooks and native auth. We also fixed
- Fix SSO re-auth page
- Fix blank verify email page
- Resolve git refresh regression
-- Fix missing "Run on Merge" button in Job creation/edit form-
+- Fix missing "Run on Merge" button in Job creation/edit form-
- Warn users they have unsaved changes
- Updates test command suggestions and regex for common action suggestions
- Updates order of stylesheet import to fix missing border bug
@@ -243,7 +243,7 @@ Exciting things coming down the pipe - ongoing enhancements to the command bar e
## dbt Cloud v1.1.24 (April 14, 2021)
-Phew! As our company grows, so too does our changelog! Look at all these! The big chunks you'll see here are related to some ongoing in-IDE work, focused on the command bar experience, as well as some partner & connection work (see the Gits, Databricks, and so forth), and of course ongoing longer-term bets around metadata!
+Phew! As our company grows, so too does our changelog! Look at all these! The big chunks you'll see here are related to some ongoing in-IDE work, focused on the command bar experience, as well as some partner & connection work (see the Gits, Databricks, and so forth), and of course ongoing longer-term bets around metadata!
#### Enhancements
@@ -282,7 +282,7 @@ Phew! As our company grows, so too does our changelog! Look at all these! The bi
## dbt Cloud v1.1.23 (March 31, 2021)
-Some backend work, some frontend work, some bug fixes: a nice mix for this release. A few user facing changes you may have noticed already are the persistence of dark/light mode settings across refresh (no more blinding IDE!), branches in the IDE being categorized by Active vs. Removed from Remote, and a tidier new file creation flow, with the file tree expanding to show the new file and opening a new tab to populate the said file!
+Some backend work, some frontend work, some bug fixes: a nice mix for this release. A few user facing changes you may have noticed already are the persistence of dark/light mode settings across refresh (no more blinding IDE!), branches in the IDE being categorized by Active vs. Removed from Remote, and a tidier new file creation flow, with the file tree expanding to show the new file and opening a new tab to populate the said file!
#### Enhancements
@@ -291,11 +291,11 @@ Some backend work, some frontend work, some bug fixes: a nice mix for this relea
- Upgrade to Tailwind 2.0 and FUI 0.0.5
- Allow users to create metadata tokens from the UI
- Support manually-managed group memberships
-- SSO: resolve bug w/ first & last names acting up
+- SSO: resolve bug w/ first & last names acting up
- Integrate Delighted for NPS surveys
-- Add dbt 0.19.1rc1 to Cloud
+- Add dbt 0.19.1rc1 to Cloud
- Add an account-level setting to require users to re-authenticate via SSO
-- Read-only metadata ServiceToken for Cloud
+- Read-only metadata ServiceToken for Cloud
- Persist IDE light mode / dark mode across refresh
- Categorize & order git branches
- Improve new file creation flow
@@ -336,7 +336,7 @@ Rolling out a few long-term bets to ensure that our beloved dbt Cloud does not f
## dbt Cloud v1.1.21 (March 3, 2021)
-This changelog wraps up work on what we've been calling the SQL Drawer in the IDE - some design nudges, some interface adjustments, overall a cleaner and snappier experience. If you haven't dipped into the IDE in a while it's worth taking a look! Some back-end work as well, making SSO and role based admin easier and more broadly available for Enterprise level folks, along with your usual assortment of bug squashes and iterations.
+This changelog wraps up work on what we've been calling the SQL Drawer in the IDE - some design nudges, some interface adjustments, overall a cleaner and snappier experience. If you haven't dipped into the IDE in a while it's worth taking a look! Some back-end work as well, making SSO and role based admin easier and more broadly available for Enterprise level folks, along with your usual assortment of bug squashes and iterations.
#### Enhancements
@@ -359,7 +359,7 @@ This changelog wraps up work on what we've been calling the SQL Drawer in the ID
## dbt Cloud v1.1.20 (February 17, 2021)
-Continued stability and quality of life improvements for folks with multiple accounts and projects - no longer will you have to remember the chronological order of birth of your accounts and projects, as they'll be ordered by the much easier to parse (for human brains anyway) alphabetical order. We're also shipping some experience improvements in the SQL Drawer at the bottom half of the IDE.
+Continued stability and quality of life improvements for folks with multiple accounts and projects - no longer will you have to remember the chronological order of birth of your accounts and projects, as they'll be ordered by the much easier to parse (for human brains anyway) alphabetical order. We're also shipping some experience improvements in the SQL Drawer at the bottom half of the IDE.
#### Enhancements
@@ -377,7 +377,7 @@ Continued stability and quality of life improvements for folks with multiple acc
## dbt Cloud v1.1.19 (February 3, 2021)
-The latest release of dbt (Oh Nineteen Oh) is now available for your enjoyment on dbt Cloud! We're also releasing some service token pieces here, though they're not quite ready for wide release yet. Moving forward, Oh Nineteen Oh will probably end up being the minimum version required to run the Metadata API & Metadata Toolkit, so, this is a big release!
+The latest release of dbt (Oh Nineteen Oh) is now available for your enjoyment on dbt Cloud! We're also releasing some service token pieces here, though they're not quite ready for wide release yet. Moving forward, Oh Nineteen Oh will probably end up being the minimum version required to run the Metadata API & Metadata Toolkit, so, this is a big release!
#### Enhancements
@@ -417,4 +417,3 @@ Most notable things here are around foundational work toward future feature rele
- Add DBT_CLOUD_CONTEXT environment variable
- Add logic to hide IP whitelist message for on-prem customers
- fix 0.19.0rc1 run image dependencies
-
diff --git a/website/docs/docs/dbt-versions/upgrade-core-in-cloud.md b/website/docs/docs/dbt-versions/upgrade-core-in-cloud.md
new file mode 100644
index 00000000000..f0b99499891
--- /dev/null
+++ b/website/docs/docs/dbt-versions/upgrade-core-in-cloud.md
@@ -0,0 +1,314 @@
+---
+title: "Upgrade Core version in Cloud"
+id: "upgrade-core-in-cloud"
+---
+
+## Upgrading to the latest version of dbt in Cloud
+
+In dbt Cloud, both jobs and environments are configured to use a specific version of dbt Core. The version can be upgraded at any time.
+
+### Environments
+
+Navigate to the settings page of an environment, then click **edit**. Click the **dbt Version** dropdown bar and make your selection. From this list, you can select an available version of Core to associate with this environment.
+
+
@@ -50,5 +50,5 @@ The following diagram shows the network architecture for the hosted _Multi Tenan
- **EC2**: The hosted dbt Cloud deployments leverage a cluster of [AWS EC2](https://aws.amazon.com/ec2/) worker nodes to run the dbt Cloud application.
- **EBS**: In order to store application data, dbt Cloud leverages [AWS Elastic Block Store](https://aws.amazon.com/ebs/) mounted to the EC2 instances described above.
- **EFS**: An [AWS Elastic File System](https://aws.amazon.com/efs/) is provisioned for hosted deployments to store and manage local files from the dbt Cloud IDE.
-- **S3**: [AWS Simple Storage Service (S3)](https://aws.amazon.com/s3/) is used to store dbt Cloud application logs and artifacts (such as those generated from dbt job runs).
+- **S3**: [AWS Simple Storage Service (S3)](https://aws.amazon.com/s3/) is used to store dbt Cloud application logs and artifacts (such as those generated from dbt job runs).
- **RDS**: The hosted dbt Cloud application leverages [AWS Postgres RDS](https://aws.amazon.com/rds/postgresql/) to store application information such as accounts, users, environments, etc. Note that as explained in the [Data Warehouse Interaction](#data-warehouse-interaction) section above, no data from an associated warehouse is ever stored in this database.
diff --git a/website/docs/docs/deploy/cloud-ci-job.md b/website/docs/docs/deploy/cloud-ci-job.md
new file mode 100644
index 00000000000..8b4afe0a544
--- /dev/null
+++ b/website/docs/docs/deploy/cloud-ci-job.md
@@ -0,0 +1,202 @@
+---
+title: "dbt Cloud CI job"
+id: "cloud-ci-job"
+description: "You can enable continuous integration (CI) to test every single change prior to deploying the code to production just like in a software development workflow."
+---
+
+## Overview
+
+dbt Cloud makes it easy to test every single code change you make prior to deploying that new logic into production. Once you've connected your [GitHub account](/docs/collaborate/git/connect-github), [GitLab account](/docs/collaborate/git/connect-gitlab), or [Azure DevOps account](/docs/collaborate/git/connect-azure-devops), you can configure jobs to run when new pull requests are opened against your dbt repo.
+
+dbt Cloud will build the models affected by the new pull request code change in a temp schema, which acts as a quasi-staging environment, and will also run the tests that you've written for these models as a check. When the continuous integration (CI) job completes, the run status will be shown directly in the pull request. This makes it possible to deploy new code to production with confidence.
+
+:::info Draft Pull Requests
+
+Jobs will _not_ be triggered by draft pull requests. If you would like jobs to run on each new commit, please mark your pull request as **Ready for review**.
+
+:::
+
+:::info GitLab Compatibility
+
+GitLab Webhooks are available to only GitLab users who have a paid or self-hosted GitLab account.
+
+:::
+
+:::info Common Errors
+If you previously configured your dbt project by providing a generic git URL that clones using SSH, you need to [reconfigure the project](/docs/deploy/cloud-ci-job#reconnecting-your-dbt-project-to-use-dbt-clouds-native-integration-with-github-gitlab-or-azure-devops) to connect through dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps instead.
+:::
+
+## Understanding dbt Cloud Slim CI
+When a [dbt Cloud CI job is set up](/docs/deploy/cloud-ci-job#configuring-a-dbt-cloud-ci-job), dbt Cloud will listen for webhooks from GitHub, GitLab, or Azure DevOps indicating that a new PR has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the CI job. Crucially, this run will build into a temporary schema using the prefix `dbt_cloud_pr_`. This schema isolation acts as a quasi-staging environment, so that you can see the builds resulting from the code associated with the PR's commit sha. The unique schema name can be found in the run details for the given run, as shown below.
+
+
+
+Only supported by v1.1 or newer.
+
+
+
+
+
+Only supported by v1.1 or newer.
+
+:::caution Experimental functionality
+The `source_status` selection is experimental and subject to change. During this time, ongoing improvements may limit this feature’s availability and cause breaking changes to its functionality.
+:::
+
+When a job is selected, dbt Cloud will surface the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of fresh sources. In your job commands, you can signal to dbt to run and test only on these fresher sources and their children by including the `source_status:fresher+` argument. This requires both previous and current state to have the `sources.json` artifact be available. Or plainly said, both job states need to run `dbt source freshness`.
+
+As example:
+```bash
+# Command step order
+dbt source freshness
+dbt build --select source_status:fresher+
+```
+
+
+More example commands in [Pro-tips for workflows](/guides/legacy/best-practices.md#pro-tips-for-workflows).
+
+## Limitations
+
+If your temporary PR schemas aren't dropping after a merge or close of the PR, it's likely due to the below scenarios. Open and review the toggles below for recommendations on how to resolve this:
+
+&jobId=&token=
+```
+
+views that your business intelligence tools and end users query. Before continuing, make sure you understand dbt's approach to [managing environments](managing-environments).
+Running dbt in production means setting up a system to run a _dbt job on a schedule_, rather than running dbt commands manually from the command line. Your production dbt jobs should create the tables and views that your business intelligence tools and end users query. Before continuing, make sure you understand dbt's approach to [managing environments](/docs/collaborate/environments).
-:::info dbt commands in production
+In addition to setting up a schedule, there are other considerations when setting up dbt to run in production:
-We've written a guide for the dbt commands we run in production, over on [Discourse](https://discourse.getdbt.com/t/what-are-the-dbt-commands-you-run-in-your-production-deployment-of-dbt/366).
-
-:::
-
-As well as setting up a schedule, there are a number of other things you should consider when setting up dbt to run in production, such as:
-* The complexity involved in creating a new dbt job, or editing an existing one.
-* Setting up notifications if a step within your job returns an error code (e.g. a model cannot be built, or a test fails).
+* The complexity involved in creating a new dbt job or editing an existing one.
+* Setting up notifications if a step within your job returns an error code (for example, a model can't be built or a test fails).
* Accessing logs to help debug any issues.
-* Pulling the latest version of your git repo before running dbt (i.e. continuous deployment).
-* Running your dbt project before merging code into master (i.e. continuous integration).
+* Pulling the latest version of your git repo before running dbt (continuous deployment).
+* Running your dbt project before merging code into master (continuous integration).
* Allowing access for team members that need to collaborate on your dbt project.
-## Ways to run dbt in production
-### Using dbt Cloud
-We've built [dbt Cloud](https://www.getdbt.com/signup/) from the ground up to empower data teams to easily run dbt in production. With dbt Cloud, you can:
+## Run dbt in production
+
+If you want to run dbt jobs on a schedule, you can use tools such as dbt Cloud, Airflow, Prefect, Dagster, automation server, or Cron.
+
+## dbt Cloud
+
+We've built dbt Cloud to empower data teams to easily run dbt in production. If you're interested in trying out dbt Cloud, you can [sign up for an account](https://cloud.getdbt.com/signup/).
+
+dbt Cloud enables you to:
- run your jobs on a schedule
- view logs for any historical invocation of dbt
- configure error notifications
- render your project's documentation
-If you're interested in giving dbt Cloud a spin, you can sign up for a *forever free* account [here](https://cloud.getdbt.com/signup/).
+In general, the dbt Cloud application deployment models fall into two categories: **Multi Tenant** and **Single Tenant**. These deployments are hosted on infrastructure managed by dbt Labs. Both models leverage AWS infrastructure as described in the [Architecture](/docs/deploy/architecture) section.
+
+For more information on these deployment models, refer to:
-
54.81.134.249
52.22.161.231 | ✅ | ✅ | ✅ | +| EMEA | eu-central-1 (Frankfurt) | emea.dbt.com | 3.123.45.39
3.126.140.248
3.72.153.148 | ❌ | ❌ | ✅ | +| Virtual Private dbt | Customized | Customized | Ask [Support](/guides/legacy/getting-help#dbt-cloud-support) for your IPs | ❌ | ❌ | ✅ | + + + diff --git a/website/docs/docs/dbt-cloud/deployments/single-tenant.md b/website/docs/docs/deploy/single-tenant.md similarity index 82% rename from website/docs/docs/dbt-cloud/deployments/single-tenant.md rename to website/docs/docs/deploy/single-tenant.md index 92945c50eaf..54cd764dadf 100644 --- a/website/docs/docs/dbt-cloud/deployments/single-tenant.md +++ b/website/docs/docs/deploy/single-tenant.md @@ -1,6 +1,6 @@ --- -id: single-tenant-deployment -title: Single Tenant +title: Single tenant +id: single-tenant --- The Single Tenant deployment environment provides a hosted alternative to the Multi Tenant (SaaS) dbt Cloud environment. While still managed and maintained by dbt Labs, the Single Tenant environment provides dedicated infrastructure with one or more instances of dbt Cloud that can only be accessed by a single customer. This is accomplished by spinning up all the necessary infrastructure with a re-usable Infrastructure as Code (IaC) deployment built with [Terraform](https://www.terraform.io/). The Single Tenant infrastructure lives in a dedicated AWS account and can be customized with certain configurations such as Firewall rules to limit ingress traffic or hosting in a specific AWS Region. @@ -13,5 +13,5 @@ _To learn more about setting up a dbt Cloud Single Tenant deployment, [please co For more information about the dbt Cloud Single Tenant deployment see the below. -- [Application Data Flows](/docs/dbt-cloud/deployments/deployment-architecture#application-data-flows) -- [Hosted Network Architecture](/docs/dbt-cloud/deployments/deployment-architecture#hosted-network-architecture) +- [Application Data Flows](/docs/deploy/architecture#application-data-flows) +- [Hosted Network Architecture](/docs/deploy/architecture#hosted-network-architecture) diff --git a/website/docs/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness.md b/website/docs/docs/deploy/source-freshness.md similarity index 80% rename from website/docs/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness.md rename to website/docs/docs/deploy/source-freshness.md index ca0073d30c4..5056e258eda 100644 --- a/website/docs/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness.md +++ b/website/docs/docs/deploy/source-freshness.md @@ -1,22 +1,23 @@ --- title: "Source freshness" -id: "cloud-snapshotting-source-freshness" +id: "source-freshness" description: "Validate that data freshness meets expectations and alert if stale." --- ## Data Source Freshness -dbt Cloud provides a helpful interface around dbt's [source data freshness](using-sources#snapshotting-source-data-freshness) calculations. When a dbt Cloud job is configured to snapshot source data freshness, dbt Cloud will render a user interface showing you the state of the most recent snapshot. This interface is intended to help you determine if your source data freshness is meeting the SLAs that you've defined for your organization. +dbt Cloud provides a helpful interface around dbt's [source data freshness](/docs/build/sources#snapshotting-source-data-freshness) calculations. When a dbt Cloud job is configured to snapshot source data freshness, dbt Cloud will render a user interface showing you the state of the most recent snapshot. This interface is intended to help you determine if your source data freshness is meeting the SLAs that you've defined for your organization. -
- - **v0.21.0:** Renamed `dbt source snapshot-freshness` to `dbt source freshness`. If using an older version of dbt, the command is `snapshot-freshness`.
+ - **v0.21.0:** Renamed `dbt source snapshot-freshness` to `dbt source freshness`. If using an older version of dbt, the command is `snapshot-freshness`.
+ To have dbt Cloud display data source freshness as a rendered user interface, you will still need to use the pre-v0.21 syntax of `dbt source snapshot-freshness`.
diff --git a/website/docs/docs/running-a-dbt-project/using-the-cli.md b/website/docs/docs/get-started/about-the-cli.md
similarity index 79%
rename from website/docs/docs/running-a-dbt-project/using-the-cli.md
rename to website/docs/docs/get-started/about-the-cli.md
index a12c50cde40..cc3639d933f 100644
--- a/website/docs/docs/running-a-dbt-project/using-the-cli.md
+++ b/website/docs/docs/get-started/about-the-cli.md
@@ -1,5 +1,6 @@
---
-title: "Using the CLI"
+title: "About the CLI"
+id: "about-the-cli"
---
dbt ships with a Command Line Interface (CLI) for running your dbt project. This way of running dbt a dbt project is free and open source.
@@ -9,6 +10,8 @@ To use the CLI, your workflow generally looks like:
* **Run your project from the command line:**
* macOS ships with a default Terminal program, however you can also use iTerm or the command line prompt within a code editor to execute dbt commands
+The CLI is also available for dbt Cloud. Additional components must be installed for the CLI to communicate via dbt Cloud APIs. For more information, visit the [dbt Cloud CLI GitHub repository](https://github.com/data-mie/dbt-cloud-cli)
+
:::info How we set up our computers for working on dbt projects
We've written a [guide](https://discourse.getdbt.com/t/how-we-set-up-our-computers-for-working-on-dbt-projects/243) for our recommended setup when running dbt projects using the CLI.
diff --git a/website/docs/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-your-database.md b/website/docs/docs/get-started/connect-your-database.md
similarity index 55%
rename from website/docs/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-your-database.md
rename to website/docs/docs/get-started/connect-your-database.md
index 2ef4726b44b..656288be68a 100644
--- a/website/docs/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-your-database.md
+++ b/website/docs/docs/get-started/connect-your-database.md
@@ -1,44 +1,43 @@
---
-title: "Connecting your database"
-id: "connecting-your-database"
+title: "Connect your database"
+id: "connect-your-database"
---
+You can connect to your database in dbt Cloud by clicking the gear in the top right and then selecting **Account Settings**. From the Account Settings page, click **+ New Project**.
+
## IP Restrictions
-dbt Cloud will always connect to your warehouse from the following IP addresses.
-Be sure to allow traffic from these IPs in your firewall, and include them in
-any database grants.
+dbt Cloud will always connect to your data platform from the IP addresses specified in the [Regions](/docs/deploy/regions) page.
-- 52.45.144.63
-- 54.81.134.249
-- 52.22.161.231
+Be sure to allow traffic from these IPs in your firewall, and include them in any database grants.
-- `54.81.134.249` and `52.22.161.231` were added in November, 2020
+- November 2020 — add the IPs `54.81.134.249` and `52.22.161.231`
+- September 2022 — Add EMEA IPs
-Allowing these IP addresses only enables the connection to your data warehouse. However, you might want to send API requests from your restricted network to the dbt Cloud API. For example, you could use the API to send a POST request that [triggers a job to run](https://docs.getdbt.com/dbt-cloud/api-v2#operation/triggerRun). Using the dbt Cloud API requires that you allow the `cloud.getdbt.com` subdomain. For more on the dbt Cloud architecture, see "[Deployment architecture](deployment-architecture)."
+Allowing these IP addresses only enables the connection to your
@@ -24,7 +25,18 @@ profile: 'jaffle_shop'
dbt then checks your `profiles.yml` file for a profile with the same name. A profile contains all the details required to connect to your data warehouse.
-This file generally lives outside of your dbt project to avoid sensitive credentials being check in to version control. By default, dbt expects the `profiles.yml` file to be located in the `~/.dbt/` directory.
+
+
+By default, dbt expects the `profiles.yml` file to be located in the `~/.dbt/` directory.
+
+
+
+
+dbt will search the current working directory for the `profiles.yml` file and will default to the `~/.dbt/` directory if not found.
+
+
+
+This file generally lives outside of your dbt project to avoid sensitive credentials being checked in to version control, but `profiles.yml` can be safely checked in when [using environment variables](#advanced-using-environment-variables) to load sensitive credentials.
@@ -59,7 +71,7 @@ A profile consists of _targets_, and a specified _default target_.
Each _target_ specifies the type of warehouse you are connecting to, the credentials to connect to the warehouse, and some dbt-specific configurations.
-The credentials you need to provide in your target varies across warehouses – sample profiles for each supported warehouse are available in the [Supported Adapters](available-adapters) section.
+The credentials you need to provide in your target varies across warehouses &mdash sample profiles for each supported warehouse are available in the [Supported Data Platforms](supported-data-platforms) section.
**Pro Tip:** You may need to surround your password in quotes if it contains special characters. More details [here](https://stackoverflow.com/a/37015689/10415173).
@@ -85,11 +97,11 @@ Use the [debug](debug) command to check whether you can successfully connect to
## Understanding targets in profiles
-dbt supports multiple targets within one profile to encourage the use of separate development and production environments as discussed in [Managing Environments](managing-environments).
+dbt supports multiple targets within one profile to encourage the use of separate development and production environments as discussed in [Managing Environments](/docs/collaborate/environments).
A typical profile for an analyst using dbt locally will have a target named `dev`, and have this set as the default.
-You may also have a `prod` target within your profile, which creates the objects in your production schema. However, since it's often desirable to perform production runs on a schedule, we recommend deploying your dbt project on a separate machine to your local machine. As such, most dbt users will only have a dev target in their profile on their local machine.
+You may also have a `prod` target within your profile, which creates the objects in your production schema. However, since it's often desirable to perform production runs on a schedule, we recommend deploying your dbt project to a separate machine other than your local machine. Most dbt users only have a `dev` target in their profile on their local machine.
If you do have multiple targets in your profile, and want to use a target other than the default, you can do this using the `--target` option when issuing a dbt command.
@@ -125,7 +137,7 @@ In development, a pattern we’ve found to work well is to name the schema in yo
Note that there’s no need to create your target schema beforehand – dbt will check if the schema already exists when it runs, and create it if it doesn’t.
-While the target schema represents the default schema that dbt will use, it may make sense to split your models into separate schemas, which can be done by using [custom schemas](using-custom-schemas).
+While the target schema represents the default schema that dbt will use, it may make sense to split your models into separate schemas, which can be done by using [custom schemas](/docs/build/custom-schemas).
## Understanding threads
@@ -143,7 +155,25 @@ You can use a different number of threads than the value defined in your target
## Advanced: Customizing a profile directory
-By default, dbt expects your `profiles.yml` file to be located in the `~/.dbt/` directory. To check the expected location of your `profiles.yml` file for your installation of dbt, you can run the following:
+The parent directory for `profiles.yml` is determined using the following precedence:
+
+
+
+1. `--profiles-dir` option
+1. `DBT_PROFILES_DIR` environment variable
+1. `~/.dbt/` directory
+
+
+
+
+1. `--profiles-dir` option
+1. `DBT_PROFILES_DIR` environment variable
+1. current working directory
+1. `~/.dbt/` directory
+
+
+
+To check the expected location of your `profiles.yml` file for your installation of dbt, you can run the following:
```bash
$ dbt debug --config-dir
@@ -152,13 +182,13 @@ To view your profiles.yml file, run:
open /Users/alice/.dbt
```
-You may want to have your `profiles.yml` file stored in a different directory – for example, if you are using environment variables to load your credentials, you might choose to include this file in your version controlled dbt project, and direct dbt to load the file from there.
+You may want to have your `profiles.yml` file stored in a different directory than `~/.dbt/` – for example, if you are [using environment variables](#advanced-using-environment-variables) to load your credentials, you might choose to include this file in the root directory of your dbt project.
Note that the file always needs to be called `profiles.yml`, regardless of which directory it is in.
-There are two ways to direct dbt to a different location for your `profiles.yml` file:
+There are multiple ways to direct dbt to a different location for your `profiles.yml` file:
-**1. Use the `--profiles-dir` option when executing a dbt command**
+### 1. Use the `--profiles-dir` option when executing a dbt command
This option can be used as follows:
```
@@ -167,13 +197,12 @@ $ dbt run --profiles-dir path/to/directory
If using this method, the `--profiles-dir` option needs to be provided every time you run a dbt command.
-**2. Use the `DBT_PROFILES_DIR` environment variable to change the default location**
+### 2. Use the `DBT_PROFILES_DIR` environment variable to change the default location
Specifying this environment variable overrides the directory that dbt looks for your `profiles.yml` file in. You can specify this by running:
```
$ export DBT_PROFILES_DIR=path/to/directory
```
-If the `--profiles-dir` option is used in a dbt command, it will take precedence over this environment variable.
## Advanced: Using environment variables
-Credentials can either be placed directly into the `profiles.yml` file, or they can be loaded from environment variables. This is especially useful for production deployments of dbt. You can find more information about using environment variables [here](env_var).
+Credentials can be placed directly into the `profiles.yml` file or loaded from environment variables. Using environment variables is especially useful for production deployments of dbt. You can find more information about environment variables [here](env_var).
diff --git a/website/docs/docs/get-started/dbt-cloud-features.md b/website/docs/docs/get-started/dbt-cloud-features.md
new file mode 100644
index 00000000000..c5963e53c4f
--- /dev/null
+++ b/website/docs/docs/get-started/dbt-cloud-features.md
@@ -0,0 +1,47 @@
+---
+title: "dbt Cloud features"
+id: "dbt-cloud-features"
+---
+
+
+## dbt Cloud IDE
+
+
- Unsaved **(•)** — The IDE detects unsaved changes to your file/folder
- Modification **(M)** — The IDE detects a modification of existing files/folders
- Untracked **(U)** — The IDE detects changes made to new files or renamed files +| **Build, test, and run code** | Build, test, and run your project with a button click or by using the Cloud IDE command bar. +| **Drag and drop** | Drag and drop files located in the file explorer, and use the file breadcrumb on the top of the IDE for quick, linear navigation. Access adjacent files in the same file by right clicking on the breadcrumb file. +| **Organize tabs** | You can:
- Move your tabs around to reorganize your work in the IDE
- Right-click on a tab to view and select a list of actions to take
- Close multiple, unsaved tabs to batch save your work +| **Multiple selections** | You can make multiple selections for small and simultaneous edits. The below commands are a common way to add more cursors and allow you to insert cursors below or above with ease.
- Option-Command-Down arrow
- Option-Command-Up arrow
- Press Option and click on an area +| **Formatting** | Format your files with a click of a button, powered by [sqlfmt](http://sqlfmt.com/). +| **Git diff view** | Ability to see what has been changed in a file before you make a pull request. +| **dbt autocomplete** | There are four new types of autocomplete features to help you develop faster:
- Use `ref` to autocomplete your model names
- Use `source` to autocomplete your source name + table name
- Use `macro` to autocomplete your arguments
- Use `env var` to autocomplete env var +| **Dark mode** | Use dark mode in the Cloud IDE for a great viewing experience in low-light environments. + + +## Related docs + +- [dbt Cloud tips](/docs/get-started/dbt-cloud-tips) +- [Develop in the Cloud](docs/get-started/develop-in-the-cloud) +- [Guides](/docs/get-started/getting-started/overview) diff --git a/website/docs/docs/get-started/dbt-cloud-tips.md b/website/docs/docs/get-started/dbt-cloud-tips.md new file mode 100644 index 00000000000..687c6c97643 --- /dev/null +++ b/website/docs/docs/get-started/dbt-cloud-tips.md @@ -0,0 +1,95 @@ +--- +title: "dbt Cloud tips" +id: dbt-cloud-tips +description: "Check out any dbt Cloud and IDE-related tips." +sidebar_label: "dbt Cloud tips" +--- + +# dbt Cloud tips + +The Cloud IDE provides keyboard shortcuts, features, and development tips to help you work faster and be more productive. Use this Cloud IDE cheat sheet to help you quickly reference some common operations. + +## Most popular + +**Organize tabs** + +You can move your tabs around to reorganize your work in the IDE. You can also right-click on a tab to close multiple tabs or view and select a list of actions to take. Right-click in the editor to access other options, including text wrap. + +**Drag and drop** + +You can also drag and drop files in the file tree or in the editor. Use the file breadcrumb on the top of the IDE for quicker navigation. You can access adjacent files in the same file by right-clicking on the breadcrumb file. + +**Search across files** + +You can quickly search over all files in the IDE on your current project. To search, open the search bar by pressing Command-O to find text across all files in your current project and write your filename. You can view the results under the search text, which are grouped into files containing the match. You can click on the results to view them in the IDE. + +**Command bar and status** + +You can run commands from the command bar at the bottom of the IDE or by using the **Build** button. Use the [rich model selection syntax](/docs/reference/node-selection/syntax) to run [dbt commands](/docs/reference/dbt-commands) directly within dbt Cloud. You can also view the history, status, and logs of previous runs by clicking **Runs**. + +The status icon on the lower right corner of the IDE gives you an indicator of the health of your project. You can identify errors by clicking on the status icon for more details or by clicking **Restart the IDE**. + +**Find and replace** + +Press Command-F or Control-F to open the find-and-replace bar in the upper right corner of the current file in the IDE. The IDE highlights your search results in the current file and code outline. You can use the up and down arrows to see the match highlighted in the current file when there are multiple matches. To replace the text with something else, use the left arrow. + +**DAG in the IDE** + +Double-click a node in the directed acyclic graph (DAG) to open that file in a new tab. Click **Expand** on the upper right side of the DAG and use node selection syntax (`select` or `exclude`) to view a subset of your DAG. + +## IDE Keyboard shortcuts + +There are default keyboard shortcuts that can help make development more productive and easier for everyone. + +- Command-O or Control-O to select a file to open +- Command-P or Control-P to see command palette +- Hold Option-click-on-an-area to select multiple lines and perform a multi-edit. You can also press Command-E to perform this operation on the command line. +- Press Fn-F1 to view a list of the other editor shortcuts +- Command-Enter or Control-Enter to Preview your code +- Command-Shift-Enter or Control-Shift-Enter to Compile +- Highlight a portion of code and use the above shortcuts to Preview or Compile code +- Enter two underscores (__) in the IDE to reveal a list of dbt functions + +## Multiple selections + +You can make multiple selections for small and simultaneous edits. The below commands are a common way to add more cursors and allow you to insert cursors below or above with ease. + +- Option-Command-Down arrow +- Option-Command-Up arrow +- Press Option and click on an area + +## Package tips + +- Use the [dbt_codegen](https://hub.getdbt.com/dbt-labs/codegen/latest/) package to help you generate YML files for your models and sources and SQL files for your staging models. +- The [dbt_utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) package contains macros useful for daily development. For example, `date_spine` generates a table with all dates between the ones provided as parameters. +- The [dbt_project_evaluator](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest) package compares your dbt project against a list of our best practices and provides suggestions and guidelines on how to update your models. +- The [dbt_expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest) package contains many tests beyond those built into dbt Core. +- The [dbt_audit_helper](https://hub.getdbt.com/#:~:text=adwords-,audit_helper,-codegen) package lets you compare the output of 2 queries. Use it when refactoring existing logic to ensure that the new results are identical. +- The [dbt_artifacts](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest) package saves information about your dbt runs directly to your data platform so that you can track the performance of models over time. +- The [dbt_meta_testing](https://hub.getdbt.com/tnightengale/dbt_meta_testing/latest) package checks that your dbt project is sufficiently tested and documented. + +## Advanced tips + +- Use your folder structure as your primary selector method. `dbt build --select marts.marketing` is simpler and more resilient than relying on tagging every model. +- Think about jobs in terms of build cadences and SLAs. Run models that have hourly, daily, or weekly build cadences together. +- Use the [where config](/docs/reference/resource-configs/where) for tests to test an assertion on a subset of records. +- [store_failures](/docs/reference/resource-configs/store_failures) lets you examine records that cause tests to fail, so you can either repair the data or change the test as needed. +- Use [severity](/docs/reference/resource-configs/severity) thresholds to set an acceptable number of failures for a test. +- Use [incremental_strategy](/docs/build/incremental-models#about-incremental_strategy) in your incremental model config to implement the most effective behavior depending on the volume of your data and reliability of your unique keys. +- Set `vars` in your `dbt_project.yml` to define global defaults for certain conditions, which you can then override using the `--vars` flag in your commands. +- Use [for loops](/docs/get-started/learning-more/using-jinja#use-a-for-loop-in-models-for-repeated-sql) in Jinja to [DRY](https://docs.getdbt.com/terms/dry) up repetitive logic, such as selecting a series of columns that all require the same transformations and naming patterns to be applied. +- Instead of relying on post-hooks, use the [grants config](/docs/reference/resource-configs/grants) to apply permission grants in the warehouse resiliently. +- Define [source-freshness](/docs/build/sources#snapshotting-source-data-freshness) thresholds on your sources to avoid running transformations on data that has already been processed. +- Use the `+` operator on the left of a model `dbt build --select +model_name` to run a model and all of its upstream dependencies. Use the `+` operator on the right of the model `dbt build --select model_name+` to run a model and everything downstream that depends on it. +- Use `dir_name` to run all models in a package or directory. +- Use the `@` operator on the left of a model in a non-state-aware CI setup to test it. This operator runs all of a selection’s parents and children, and also runs the parents of its children, which in a fresh CI schema will likely not exist yet. +- Use the [--exclude flag](/docs/reference/node-selection/exclude) to remove a subset of models out of a selection. +- Use [state and deferral](/docs/deploy/cloud-ci-job#deferral-and-state-comparison) to create a slim CI setup. +- Use the [--full-refresh](/docs/reference/commands/run#refresh-incremental-models) flag to rebuild an incremental model from scratch. +- Use [seeds](/docs/build/seeds) to create manual lookup tables, like zip codes to states or marketing UTMs to campaigns. `dbt seed` will build these from CSVs into your warehouse and make them `ref` able in your models. +- Use [target.name](/docs/build/custom-schemas#an-alternative-pattern-for-generating-schema-names) to pivot logic based on what environment you’re using. For example, to build into a single development schema while developing, but use multiple schemas in production. + +## Related docs + +- [Getting started](/docs/get-started/getting-started/overview) +- [Develop in the Cloud](/docs/get-started/develop-in-the-cloud) diff --git a/website/docs/docs/get-started/develop-in-the-cloud.md b/website/docs/docs/get-started/develop-in-the-cloud.md new file mode 100644 index 00000000000..a29e18082ed --- /dev/null +++ b/website/docs/docs/get-started/develop-in-the-cloud.md @@ -0,0 +1,184 @@ +--- +title: "Develop in the Cloud" +id: develop-in-the-cloud +description: "Develop, test, run, and build in the Cloud IDE." +sidebar_label: Develop in the Cloud +--- + +
+
+
+## Set up and access the Cloud IDE
+
+:::info📌
+
+New to dbt? Check out our [Getting Started guide](/docs/get-started/getting-started/overview) to build your first dbt project in the Cloud IDE!
+
+:::
+
+In order to start experiencing the great features of the Cloud IDE, you need to first set up your **Development environment** and **Development credentials.**
+
+If you’re new to dbt, you will automatically add this during the project setup. However, if you have an existing dbt Cloud account, you may need to create a development environment and credentials manually to use the Cloud IDE.
+
+Review the steps below to set up your development environment and credentials:
+
+
+**Development environment**
+
+1. Create a development environment and choose **Deploy** and then **Environments** from the top left. Click **Create Environment**.
+
+
Compile
Build | This is where you can preview, compile or build your dbt project, as well as see the results and the DAG. | +| 7. | Lineage tab | You can see how models are used as building blocks from left to right to transform your data from raw sources into cleaned-up modular derived pieces and final outputs on the far right of the DAG. You can access files in the **Lineage** tab by double-clicking on a particular model. Expand the DAG into fullscreen to view the DAG view differently. Note: The default view is `+model+`, however, you can change it to `2+model+2`. | +| 8. | Change branches and view documentation | Change branches in fewer clicks and focus on developing. You can generate and view your [documentation](/docs/collaborate/build-and-view-your-docs) for your dbt project in real time. You can inspect and verify what your project's documentation will look like before you deploy your changes to production.| +| 9. | File state indicators | The file state indicators will indicate and track any action or changed made in your project. The indicators **M, U, and •** appear to the right of your file or folder name, and also under the **Changes** section. | +| 10. | Format button | This is where you can format your dbt project code. The new **Format** button formats your file and is powered by [sqlfmt](http://sqlfmt.com/).| + +## Build, compile, and run projects + +You can *build*, *compile*, *run* *, and test* dbt projects by using the command bar. The Cloud IDE will update in real time when you run models, tests, seeds, and operations. + +If a model or test fails, you can review the logs to find and fix the issue. + +You can also use dbt's [rich model selection syntax](/reference/node-selection/syntax) to [run dbt commands](/reference/dbt-commands) directly within dbt Cloud. + +` takes longer to run, and will always install the latest compatible versions of every dependency.
+
+You might also be able to use Docker to install and develop locally if you don't have a Python environment set up. Note that running dbt in this manner can be significantly slower if your operating system differs from the system that built the Docker image. If you're a frequent local developer, we recommend that you install dbt Core via [Homebrew](/docs/get-started/homebrew-install) or [pip](/docs/get-started/pip-install) instead.
+
+### Prerequisites
+* You've installed Docker. For more information, see the [Docker](https://docs.docker.com/) site.
+* You understand which database adapter(s) you need. For more information, see [About dbt adapters](/docs/get-started/installation#about-dbt-adapters).
+* You understand how dbt Core is versioned. For more information, see [About dbt Core versions](core-versions).
+* You have a general understanding of the dbt, dbt workflow, developing locally in the command line interface (CLI). For more information, see [About dbt](/docs/introduction#how-do-i-use-dbt).
+
+### Install a dbt Docker image from Github Packages
+
+Offical dbt docker images are hosted as [packages in the `dbt-labs` GitHub organization](https://github.com/orgs/dbt-labs/packages?visibility=public). We maintain images and tags for every version of every database adapter, as well as two tags that update as new versions as released:
+- `latest`: Latest overall version of dbt-core + this adapter
+- `..latest`: Latest patch of dbt-core + this adapter for `.` version family. For example, `1.1.latest` includes the latest patches for dbt Core v1.1.
+
+Install an image using the `docker pull` command:
+```
+docker pull ghcr.io/dbt-labs/:
+```
+
+### Running a dbt Docker image in a container
+
+The `ENTRYPOINT` for dbt Docker images is the command `dbt`. You can bind-mount your project to `/usr/app` and use dbt as normal:
+```
+docker run \
+--network=host \
+--mount type=bind,source=path/to/project,target=/usr/app \
+--mount type=bind,source=path/to/profiles.yml,target=/root/.dbt/ \
+ \
+ls
+```
+
+Notes:
+* Bind-mount sources _must_ be an absolute path
+* You may need to make adjustments to the docker networking setting depending on the specifics of your data warehouse or database host.
+
+### Building your own dbt Docker image
+
+If the pre-made images don't fit your use case, we also provide a [`Dockerfile`](https://github.com/dbt-labs/dbt-core/blob/main/docker/Dockerfile) and [`README`](https://github.com/dbt-labs/dbt-core/blob/main/docker/README.md) that can be used to build custom images in a variety of ways.
+
+In particular, the Dockerfile supports building images:
+- Images that all adapters maintained by dbt Labs
+- Images that install one or more third-party adapters
+- Images against another system architecture
+
+Please note that, if you go the route of building your own Docker images, we are unable to offer dedicated support for custom use cases. If you run into problems, you are welcome to [ask the community for help](getting-help) or [open an issue](oss-expectations#issues) in the `dbt-core` repository. If many users are requesting the same enhancement, we will tag the issue `help_wanted` and invite community contribution.
diff --git a/website/docs/tutorial/learning-more/getting-started-dbt-core.md b/website/docs/docs/get-started/getting-started-dbt-core.md
similarity index 83%
rename from website/docs/tutorial/learning-more/getting-started-dbt-core.md
rename to website/docs/docs/get-started/getting-started-dbt-core.md
index db20620d06d..11c0666b44d 100644
--- a/website/docs/tutorial/learning-more/getting-started-dbt-core.md
+++ b/website/docs/docs/get-started/getting-started-dbt-core.md
@@ -2,16 +2,16 @@
title: "Getting started with dbt Core"
id: getting-started-dbt-core
description: "Connecting your warehouse to dbt Core using the CLI."
-sidebar_label: "Getting started with dbt Core"
+sidebar_label: "Getting started"
---
-When you use dbt Core to work with dbt, you will be editing files locally using a code editor, and running projects using the dbt command line interface (dbt CLI). If you'd rather edit files and run projects using the web-based Integrated Development Environment (IDE), you should refer to [Getting set up in dbt Cloud](/tutorial/getting-set-up).
+When you use dbt Core to work with dbt, you will be editing files locally using a code editor, and running projects using the dbt command line interface (dbt CLI). If you'd rather edit files and run projects using the web-based Integrated Development Environment (IDE), you should refer to [Getting set up in dbt Cloud](/docs/get-started/getting-started/set-up-dbt-cloud).
## Prerequisites
* To use the dbt CLI, it's important that you know some basics of the Terminal. In particular, you should understand `cd`, `ls` and `pwd` to navigate through the directory structure of your computer easily.
-* Install dbt Core using the [installation instructions](/dbt-cli/install/overview) for your operating system.
-* Complete [Setting up (in BigQuery)](/tutorial/getting-set-up/setting-up-bigquery#setting-up) and [Loading data (BigQuery)](/tutorial/getting-set-up/setting-up-bigquery#loading-data).
+* Install dbt Core using the [installation instructions](/docs/get-started/installation) for your operating system.
+* Complete [Setting up (in BigQuery)](/docs/get-started/getting-started/getting-set-up/setting-up-bigquery#setting-up) and [Loading data (BigQuery)](/docs/get-started/getting-started/getting-set-up/setting-up-bigquery#loading-data).
* [Create a GitHub account](https://github.com/join) if you don't already have one.
## Create a starter project
@@ -25,7 +25,7 @@ The following steps use [GitHub](https://github.com/) as the Git provider for th
1. [Create a new GitHub repository](https://github.com/new) named `dbt-tutorial`.
2. Select **Public** so the repository can be shared with others. You can always make it private later.
3. Leave the default values for all other settings.
-4. Click **Create repository**.
+4. Click **Create repository**.
5. Save the commands from "…or create a new repository on the command line" to use later in [Commit your changes](#commit-your-changes).
### Create a project
@@ -87,7 +87,7 @@ To create your dbt project:
### Connect to BigQuery
-When developing locally, dbt connects to your data warehouse using a [profile](/dbt-cli/configure-your-profile), which is a yaml file with all the connection details to your warehouse.
+When developing locally, dbt connects to your
+- For events within 90 days — dbt Cloud will automatically display the 90 days selectable date range. Select **Export Selection** to download a CSV file of all the events that occurred in your organization within 90 days.
+- For events beyond 90 days — Select **Export All**. The Account Admin will receive an email link to download a CSV file of all the events that occurred in your organization.
+
+
diff --git a/website/docs/docs/dbt-cloud/access-control/cloud-seats-and-users.md b/website/docs/docs/collaborate/manage-access/cloud-seats-and-users.md
similarity index 88%
rename from website/docs/docs/dbt-cloud/access-control/cloud-seats-and-users.md
rename to website/docs/docs/collaborate/manage-access/cloud-seats-and-users.md
index 55bad32f40b..d768e7c6e57 100644
--- a/website/docs/docs/dbt-cloud/access-control/cloud-seats-and-users.md
+++ b/website/docs/docs/collaborate/manage-access/cloud-seats-and-users.md
@@ -1,6 +1,6 @@
---
title: "Seats and Users"
-id: "cloud-seats-and-users"
+id: "seats-and-users"
---
## Overview
@@ -10,7 +10,7 @@ different types of seat licenses in dbt Cloud: _Developer_ and _Read Only_.
The type of license a user is assigned controls which capabilities of dbt
Cloud the user is permitted to access. Users with a Developer license can be
-granted access to the Deployment and [Development](the-dbt-ide) functionality
+granted access to the Deployment and [Development](/docs/get-started/develop-in-the-cloud) functionality
in dbt Cloud, whereas users with Read Only licenses are intended to view the
[artifacts](docs/dbt-cloud/using-dbt-cloud/artifacts) created in a dbt Cloud account.
@@ -20,8 +20,8 @@ in dbt Cloud, whereas users with Read Only licenses are intended to view the
| Use Jobs | ✅ | ❌ |
| Manage Account | ✅ | ❌ |
| API Access | ✅ | ❌ |
-| Use [Source Freshness](cloud-snapshotting-source-freshness) | ✅ | ✅ |
-| Use [Docs](cloud-generating-documentation) | ✅ | ✅ |
+| Use [Source Freshness](/docs/deploy/source-freshness) | ✅ | ✅ |
+| Use [Docs](/docs/collaborate/build-and-view-your-docs) | ✅ | ✅ |
## Included seats
@@ -37,7 +37,7 @@ dbt Cloud plan.
| Plan | Developer Seats | Read Only Seats |
| ---- | --------------- | --------------- |
| Developer (free) | 1 | 0 |
-| Team | $50/developer/mo | 50 ($0/mo) |
+| Team | $50/developer/mo | 5 ($0/mo) |
| Enterprise | Custom | Custom |
## Managing license types
@@ -66,7 +66,7 @@ change.
**Note:** This feature is only available on the Enterprise plan.
If your account is connected to an Identity Provider (IdP) for [Single Sign
-On](dbt-cloud-enterprise/sso-overview), you can automatically map IdP user
+On](/docs/collaborate/manage-access/sso-overview), you can automatically map IdP user
groups to specific license types in dbt Cloud. To configure license mappings,
navigate to the Account Settings > Team > License Mappings page. From
here, you can create or edit SSO mappings for both Read Only and Developer
@@ -96,5 +96,5 @@ Usage notes:
## Granular permissioning
The dbt Cloud Enterprise plan supports Role-Based access controls for
-configuring granular in-app permissions. See [access control](access-control-overview)
+configuring granular in-app permissions. See [access control](/docs/collaborate/manage-access/about-access)
for more information on Enterprise permissioning.
diff --git a/website/docs/docs/dbt-cloud/access-control/enterprise-permissions.md b/website/docs/docs/collaborate/manage-access/enterprise-permissions.md
similarity index 65%
rename from website/docs/docs/dbt-cloud/access-control/enterprise-permissions.md
rename to website/docs/docs/collaborate/manage-access/enterprise-permissions.md
index 1e81bd58d13..7a0031d3c7a 100644
--- a/website/docs/docs/dbt-cloud/access-control/enterprise-permissions.md
+++ b/website/docs/docs/collaborate/manage-access/enterprise-permissions.md
@@ -1,7 +1,7 @@
---
-title: "Enterprise Permissions"
+title: "Enterprise permissions"
id: "enterprise-permissions"
-description: "Permission sets for Enterprise plans."
+description: "Permission sets for Enterprise plans."
---
:::info Enterprise Feature
@@ -15,12 +15,14 @@ If you're interested in learning more about an Enterprise plan, contact us at sa
The dbt Cloud Enterprise plan supports a number of pre-built permission sets to
help manage access controls within a dbt Cloud account. See the docs on [access
-control](access-control-overview) for more information on Role-Based access
+control](/docs/collaborate/manage-access/about-access) for more information on Role-Based access
control (RBAC).
## Permission Sets
-The following permission sets are available for assignment in dbt Cloud Enterprise accounts. They can be granted to dbt Cloud groups, where users belong. A dbt Cloud group can be associated with more than one permission set.
+The following permission sets are available for assignment in dbt Cloud Enterprise accounts. They
+can be granted to dbt Cloud groups which are then in turn granted to users. A dbt Cloud group
+can be associated with more than one permission sets.
### Account Admin
@@ -36,18 +38,38 @@ Account Admins have unrestricted access to dbt Cloud accounts. Users with Accoun
- Create, delete, and modify Jobs
- Create, delete, and modify Groups
- Create, delete, and modify Group Memberships
-- Manage notification settings
+- Manage Notification Settings
- Manage account-level [artifacts](dbt-cloud/using-dbt-cloud/artifacts)
- View and modify Account Settings
- Use the IDE
- Run and cancel jobs
+### Project Creator
+- **Has permissions on:** Authorized projects, account-level settings
+- **License restrictions:** must have a developer license
+
+Project Creators have write and read-only access to dbt Cloud accounts, but do not have the permissions required to modify SSO settings and account integrations.
+
+Users with Project Creator permissions can:
+
+- View Account Settings
+- View and modify project users
+- Create, delete and modify all projects in an account
+- Create, delete, and modify Repositories
+- Create, delete, and modify Connections
+- Create, delete, and modify Environments
+- Create, delete, and modify Jobs
+- Use the IDE
+- Run and cancel jobs
+- View Groups
+- View Notification Settings
+
### Account Viewer
- **Has permissions on:** Authorized projects, account-level settings
- **License restrictions:** must have a developer license
-Account Viewers have read only access to dbt Cloud accounts. Users with Account Viewer permissions can:
+Account Viewers have read only access to dbt Cloud accounts. Users with Account Viewer permissions can:
- View all projects in an account
- View Account Settings
- View Repositories
@@ -56,7 +78,7 @@ Account Viewers have read only access to dbt Cloud accounts. Users with Account
- View Jobs
- View Groups
- View Group Memberships
-- View notification settings
+- View Notification Settings
- View account-level artifacts
### Admin
@@ -113,6 +135,7 @@ Team Admins can perform the following actions in projects they are assigned to:
Job Admins can perform the following actions in projects they are assigned to:
- View, edit, and create environments
+- View connections
- Trigger runs
- View historical runs
@@ -142,6 +165,7 @@ Developers can perform the following actions in projects they are assigned to:
Analysts can perform the following actions in projects they are assigned to:
- Use the IDE
- Configure personal developer credentials
+- View connections
- View environments
- View job definitions
- View historical runs
@@ -161,9 +185,21 @@ Stakeholders can perform the following actions in projects they are assigned to:
+✅ Use: `your-company-name`
+❌ Avoid: `Your-Company-Name` +4. Under the **Configuration tab**, input the following values: + + - **RelayState:** `
-Account Admin service tokens have full `read + write` access to an account, so please use them with caution. A Team plan refers to this permission set as an "Owner role." For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#account-admin). +Account Admin service tokens have full `read + write` access to an account, so please use them with caution. A Team plan refers to this permission set as an "Owner role." For more on these permissions, see [Account Admin](/docs/collaborate/manage-access/enterprise-permissions#account-admin). **Metadata Only**
Metadata only service tokens can authorize requests to the metadata API. @@ -43,47 +43,47 @@ Metadata only service tokens can authorize requests to the metadata API. Job admin service tokens can authorize requests for viewing, editing, and creating environments, triggering runs, and viewing historical runs. **Member**
-Member service tokens can authorize requests for viewing and editing resources, triggering runs, and inviting members to the account. Tokens assigned the Member permission set will have the same permissions as a Member user. For more information about Member users, see "[Self-service permissions](/dbt-cloud/access-control/self-service-permissions)". +Member service tokens can authorize requests for viewing and editing resources, triggering runs, and inviting members to the account. Tokens assigned the Member permission set will have the same permissions as a Member user. For more information about Member users, see "[Self-service permissions](/docs/collaborate/manage-access/self-service-permissions)". **Read-only**
Read-only service tokens can authorize requests for viewing a read-only dashboard, viewing generated documentation, and viewing source freshness reports. ### Enterprise plans using service account tokens -The following permissions can be assigned to a service account token on an Enterprise plan. For more details about these permissions, see "[Enterprise permissions](/docs/dbt-cloud/access-control/enterprise-permissions)." +The following permissions can be assigned to a service account token on an Enterprise plan. For more details about these permissions, see "[Enterprise permissions](/docs/collaborate/manage-access/enterprise-permissions)." **Account Admin**
-Account Admin service tokens have full `read + write` access to an account, so please use them with caution. For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#account-admin). +Account Admin service tokens have full `read + write` access to an account, so please use them with caution. For more on these permissions, see [Account Viewer](/docs/collaborate/manage-access/enterprise-permissions#account-admin). **Metadata Only**
Metadata only service tokens can authorize requests to the metadata API. **Job Admin**
-Job Admin service tokens can authorize request for viewing, editing, and creating environments, triggering runs, and viewing historical runs. For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#job-admin). +Job Admin service tokens can authorize request for viewing, editing, and creating environments, triggering runs, and viewing historical runs. For more on these permissions, see [Account Viewer](/docs/collaborate/manage-access/enterprise-permissions#job-admin). **Account Viewer**
-Account Viewer service tokens have read only access to dbt Cloud accounts. For more on these permissions, see [Account Viewer](docs/dbt-cloud/access-control/enterprise-permissions#account-viewer) on the Enterprise Permissions page. +Account Viewer service tokens have read only access to dbt Cloud accounts. For more on these permissions, see [Account Viewer](/docs/collaborate/manage-access/enterprise-permissions#account-viewer) on the Enterprise Permissions page. **Admin**
-Admin service tokens have unrestricted access to projects in dbt Cloud accounts. You have the option to grant that permission all projects in the account or grant the permission only on specific projects. For more on these permissions, see [Admin Service](docs/dbt-cloud/access-control/enterprise-permissions#admin-service) on the Enterprise Permissions page. +Admin service tokens have unrestricted access to projects in dbt Cloud accounts. You have the option to grant that permission all projects in the account or grant the permission only on specific projects. For more on these permissions, see [Admin Service](/docs/collaborate/manage-access/enterprise-permissions#admin-service) on the Enterprise Permissions page. **Git Admin**
-Git admin service tokens have all the permissions listed in [Git admin](/docs/dbt-cloud/access-control/enterprise-permissions#git-admin) on the Enterprise Permissions page. +Git admin service tokens have all the permissions listed in [Git admin](/docs/collaborate/manage-access/enterprise-permissions#git-admin) on the Enterprise Permissions page. **Database Adminn**
-Database admin service tokens have all the permissions listed in [Database admin](/docs/dbt-cloud/access-control/enterprise-permissions#database-admin) on the Enterprise Permissions page. +Database admin service tokens have all the permissions listed in [Database admin](/docs/collaborate/manage-access/enterprise-permissions#database-admin) on the Enterprise Permissions page. **Team Admin**
-Team admin service tokens have all the permissions listed in [Team admin](/docs/dbt-cloud/access-control/enterprise-permissions#team-admin) on the Enterprise Permissions page. +Team admin service tokens have all the permissions listed in [Team admin](/docs/collaborate/manage-access/enterprise-permissions#team-admin) on the Enterprise Permissions page. **Job Viewer**
-Job viewer admin service tokens have all the permissions listed in [Job viewer](/docs/dbt-cloud/access-control/enterprise-permissions#job-viewer) on the Enterprise Permissions page. +Job viewer admin service tokens have all the permissions listed in [Job viewer](/docs/collaborate/manage-access/enterprise-permissions#job-viewer) on the Enterprise Permissions page. **Developer**
-Developer service tokens have all the permissions listed in [Developer](/docs/dbt-cloud/access-control/enterprise-permissions#developer) on the Enterprise Permissions page. +Developer service tokens have all the permissions listed in [Developer](/docs/collaborate/manage-access/enterprise-permissions#developer) on the Enterprise Permissions page. **Analyst**
-Analyst admin service tokens have all the permissions listed in [Analyst](/docs/dbt-cloud/access-control/enterprise-permissions#analyst) on the Enterprise Permissions page. +Analyst admin service tokens have all the permissions listed in [Analyst](/docs/collaborate/manage-access/enterprise-permissions#analyst) on the Enterprise Permissions page. **Stakeholder**
-Stakeholder service tokens have all the permissions listed in [Stakeholder](/docs/dbt-cloud/access-control/enterprise-permissions#stakeholder) on the Enterprise Permissions page. +Stakeholder service tokens have all the permissions listed in [Stakeholder](/docs/collaborate/manage-access/enterprise-permissions#stakeholder) on the Enterprise Permissions page. diff --git a/website/docs/docs/dbt-cloud/dbt-cloud-api/user-tokens.md b/website/docs/docs/dbt-cloud-apis/user-tokens.md similarity index 100% rename from website/docs/docs/dbt-cloud/dbt-cloud-api/user-tokens.md rename to website/docs/docs/dbt-cloud-apis/user-tokens.md diff --git a/website/docs/docs/dbt-cloud/access-control/self-service-permissions.md b/website/docs/docs/dbt-cloud/access-control/self-service-permissions.md deleted file mode 100644 index e9a89281a28..00000000000 --- a/website/docs/docs/dbt-cloud/access-control/self-service-permissions.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -title: "Self-Service Permissions" -id: "self-service-permissions" ---- - -## Overview - -dbt Cloud supports two different permission sets to manage permissions for self-service accounts: **Member** and **Owner**. - -The permissions afforded to each role are described below: - -| Action | Member | Owner | -| ------ | ------ | ----- | -| View and edit resources | ✅ | ✅ | -| Trigger runs | ✅ | ✅ | -| Access the IDE | ✅ | ✅ | -| Invite Members to the account | ✅ | ✅ | -| Manage billing | ❌ | ✅ | -| Manage team permissions | ❌ | ✅ | -| Invite Owners to the account | ❌ | ✅ | - -## Read Only vs. Developer License Types - -Users configured with Read Only license types will experience a restricted set of permissions in dbt Cloud. If a user is associated with a _Member_ permission set and a Read Only seat license, then they will only have access to what a Read-Only seat allows. See [Seats and Users](cloud-seats-and-users) for more information on the impact of licenses on these permissions. - -## Owner and Member Groups in dbt Cloud Enterprise - -By default, new users are added to the Member and Owner groups when they onboard to a new dbt Cloud. Member and Owner groups are included with every new dbt Cloud account. They're used to provide access for Administrators to add additional users, groups and apply permission sets. This is necessary for account onboarding but can create confusion when initially setting up SSO and RBAC for dbt Cloud Enterprise accounts as described in the [Enterprise Permissions](enterprise-permissions) guide. Owner and Member groups are **account level** groups, so their permissions override any project-level permissions you wish to apply. - -After onboarding Administrative users and configuring RBAC/SSO groups, we recommend the following steps for onboarding users to a dbt Cloud Enterprise account. - -### Create Account Admins Group - -**Important:** Do this BEFORE proceeding to the next section - -1) Create an Account Admins group -2) Assign at least one user to the Account Admins - -The assigned user will manage future group, SSO mapping and user / group assignment. - -### Remove the Owner and Member groups - -Follow the steps below for each of the Owner and Member groups in turn: - -1) Click the Hamburger icon at the top left of the Cloud Console, then Account Settings >> Groups >>
-
- -:::info Universal change -Certain configurations in dbt_project.yml have been renamed -::: - -Existing projects will see non-breaking deprecation warnings. You can change three lines in most projects to remove the warnings: - -
-
-```yml
-model-paths: ["models"] # formerly named "source-paths"
-seed-paths: ["data"] # formerly named "data-paths"
-clean-targets:
- - "target"
- - "dbt_packages" # formerly named "dbt_modules"
-```
-
-
-
-- Do you select tests using the old names for test types? (`test_type:schema`, `test_type:data`, `--schema`, `--data`)
-- Do you have custom macro code that calls the (undocumented) global macros `column_list`, `column_list_for_create_table`, `incremental_upsert`?
-- Do you have custom scripts that parse dbt JSON artifacts?
-- (BigQuery only) Do you use dbt's legacy capabilities around ingestion-time-partitioned tables?
-
-If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-v1.0).
-
-
-
-
-Upgrading to v1.0.latest from v0.21
-- -:::info Universal change -Certain configurations in dbt_project.yml have been renamed -::: - -Existing projects will see non-breaking deprecation warnings. You can change three lines in most projects to remove the warnings: - -
-
- -- Do you select specific sources to check freshness (`dbt snapshot-freshness --select`)?
-- Do you have custom scripts that parse dbt JSON artifacts?
-- (Snowflake only) Do you have custom macros or materializations that depend on using transactions, such as statement blocks with `auto_begin=True`?
-
-If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-21-0).
-
-
-
-
-
-Upgrading to v0.21.latest from v0.20
-- -- Do you select specific sources to check freshness (`dbt snapshot-freshness --select
-
- -- Does your project define any custom schema tests? -- Does your project use `adapter.dispatch` or the `spark_utils` package? -- Do you have custom scripts that parse dbt JSON artifacts? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-20-0). - -
-
-
-
-Upgrading to v0.20.latest from v0.19
-- -- Does your project define any custom schema tests? -- Does your project use `adapter.dispatch` or the `spark_utils` package? -- Do you have custom scripts that parse dbt JSON artifacts? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-20-0). - -
-
-
-
-
-Upgrading to v0.19.latest from v0.18
--
-
-:::info Important
-
-If you have not already, you must add `config-version: 2` to your dbt_project.yml file.
-See **Upgrading to v0.17.latest from v0.16** below for more details.
-
-:::
-
-
-
-- Do you have custom scripts that parse dbt JSON artifacts?
-- Do you have any custom materializations?
-
-If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-19-0).
-
-
-
- -- Do you directly call `adapter_macro`? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-18-0). - -
-
-
-
-Upgrading to v0.18.latest from v0.17
-- -- Do you directly call `adapter_macro`? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-18-0). - -
-
-
-
-```yml
-name: my_project
-version: 1.0.0
-
-config-version: 2
-
-vars:
- my_var: 1
- another_var: true
-
-models:
- ...
-```
-
-
-
-
-
-```yml
-name: my_project
-version: 1.0.0
-
-config-version: 2
-
-vars:
- my_var: 1
- another_var: true
-
-models:
- ...
-```
-
-
-
-
-- Do you have dictionary configs in your dbt_project.yml such as `partition_by` or `persist_docs`? If yes, you need to add a preceding +.
-
-
-
-```yml
-
-models:
- my_project:
- reporting:
- +partition_by:
- field: date_day
- data_type: timestamp
-```
-
-
-If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-17-0).
-
-
-
-
-Upgrading to v0.17.latest from v0.16
--
-
-:::info Universal change
-
-You must add `config-version: 2` to your dbt_project.yml file.
-:::
-
-
-
-
-:::info Universal change
-
-`vars:` are now defined not in your `models:` but are a separate section in dbt_project.yml file.
-:::
-
-
-
-
-
- -- Do you use the custom `generate_schema_name` macro? -- Do you use `partition_by` config for BigQuery models? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-16-0). -
-
-
-Upgrading to v0.16.latest from v0.15
-- -- Do you use the custom `generate_schema_name` macro? -- Do you use `partition_by` config for BigQuery models? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-16-0). -
-
- -- Do you have a custom materialization? -- Do you have a macro that accesses `Relations` directly? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-15-0). -
-
-Upgrading to v0.15.latest from v0.14
- -- -- Do you have a custom materialization? -- Do you have a macro that accesses `Relations` directly? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-15-0). -
-
- -- Do you still use `Archives`? -- Do you use the custom `generate_schema_name` macro? -- Do you use the `—non-destructive` flag? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-14-0). -
-
-
-#### Testing your changes before upgrading
-Once you have an idea about what code changes you'll need to make, you can start implementing them. We recommend that you create a separate dbt project, `Upgrade Project`, to test your changes before making them live in your main dbt project. In your `Upgrade Project`, connect to the same repository that you use for your main dbt project, but this time, set the development environment [settings](docs/dbt-cloud/cloud-configuring-dbt-cloud/cloud-choosing-a-dbt-version) to run the latest version of dbt Core. Next check out a branch `dbt-version-upgrade`, make the appropriate updates to your project (if needed), and see if your dbt project compiles and runs with the new version of dbt in the IDE. If jumping directly to the latest version of dbt is too far of a leap for your project, try iteratively getting your project to work on each successive minor version. There are years of development and a handful of breaking changes between two distant versions of dbt (e.g. 0.14 --> 0.20). There are far fewer between two subsequent versions of dbt, which is why upgrading regularly is important.
-
-Once you have your project compiling and running on the latest version of dbt in the development environment for your `dbt-version-upgrade` branch, try replicating one of your production jobs to run off your branch's code. You can do this by creating a new deployment environment for testing, setting the custom branch to 'ON' and referencing your `dbt-version-upgrade` branch. You'll also need to set the dbt version in this environment to the latest dbt Core version.
-
-
-
-Upgrading to v0.14.latest from v0.13
-- -- Do you still use `Archives`? -- Do you use the custom `generate_schema_name` macro? -- Do you use the `—non-destructive` flag? - -If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-0-14-0). -
-
-
-
-
-
-
+
+ +:::info Universal change +Certain configurations in dbt_project.yml have been renamed +::: + +Existing projects will see non-breaking deprecation warnings. You can change three lines in most projects to remove the warnings: + +
+
+```yml
+model-paths: ["models"] # formerly named "source-paths"
+seed-paths: ["data"] # formerly named "data-paths"
+clean-targets:
+ - "target"
+ - "dbt_packages" # formerly named "dbt_modules"
+```
+
+
+
+- Do you select tests using the old names for test types? (`test_type:schema`, `test_type:data`, `--schema`, `--data`)
+- Do you have custom macro code that calls the (undocumented) global macros `column_list`, `column_list_for_create_table`, `incremental_upsert`?
+- Do you have custom scripts that parse dbt
+
+
+Upgrading to v1.0.latest from v0.21
++ +:::info Universal change +Certain configurations in dbt_project.yml have been renamed +::: + +Existing projects will see non-breaking deprecation warnings. You can change three lines in most projects to remove the warnings: + +
+
+ +- Do you select specific sources to check freshness (`dbt snapshot-freshness --select`)?
+- Do you have custom scripts that parse dbt JSON artifacts?
+- (Snowflake only) Do you have custom macros or materializations that depend on using transactions, such as statement blocks with `auto_begin=True`?
+
+If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions).
+
+
+
+
+
+Upgrading to v0.21.latest from v0.20
++ +- Do you select specific sources to check freshness (`dbt snapshot-freshness --select
+
+ +- Does your project define any custom schema tests? +- Does your project use `adapter.dispatch` or the `spark_utils` package? +- Do you have custom scripts that parse dbt JSON artifacts? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). + +
+
+
+
+Upgrading to v0.20.latest from v0.19
++ +- Does your project define any custom schema tests? +- Does your project use `adapter.dispatch` or the `spark_utils` package? +- Do you have custom scripts that parse dbt JSON artifacts? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). + +
+
+
+
+
+Upgrading to v0.19.latest from v0.18
++
+
+:::info Important
+
+If you have not already, you must add `config-version: 2` to your dbt_project.yml file.
+See **Upgrading to v0.17.latest from v0.16** below for more details.
+
+:::
+
+
+
+- Do you have custom scripts that parse dbt JSON artifacts?
+- Do you have any custom materializations?
+
+If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions).
+
+
+
+ +- Do you directly call `adapter_macro`? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). + +
+
+
+
+Upgrading to v0.18.latest from v0.17
++ +- Do you directly call `adapter_macro`? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). + +
+
+
+
+```yml
+name: my_project
+version: 1.0.0
+
+config-version: 2
+
+vars:
+ my_var: 1
+ another_var: true
+
+models:
+ ...
+```
+
+
+
+
+
+```yml
+name: my_project
+version: 1.0.0
+
+config-version: 2
+
+vars:
+ my_var: 1
+ another_var: true
+
+models:
+ ...
+```
+
+
+
+
+- Do you have dictionary configs in your dbt_project.yml such as `partition_by` or `persist_docs`? If yes, you need to add a preceding +.
+
+
+
+```yml
+
+models:
+ my_project:
+ reporting:
+ +partition_by:
+ field: date_day
+ data_type: timestamp
+```
+
+
+If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions).
+
+
+
+
+Upgrading to v0.17.latest from v0.16
++
+
+:::info Universal change
+
+You must add `config-version: 2` to your dbt_project.yml file.
+:::
+
+
+
+
+:::info Universal change
+
+`vars:` are now defined not in your `models:` but are a separate section in dbt_project.yml file.
+:::
+
+
+
+
+
+ +- Do you use the custom `generate_schema_name` macro? +- Do you use `partition_by` config for BigQuery models? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). +
+
+
+Upgrading to v0.16.latest from v0.15
++ +- Do you use the custom `generate_schema_name` macro? +- Do you use `partition_by` config for BigQuery models? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). +
+
+ +- Do you have a custom materialization? +- Do you have a macro that accesses `Relations` directly? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). +
+
+Upgrading to v0.15.latest from v0.14
+ ++ +- Do you have a custom materialization? +- Do you have a macro that accesses `Relations` directly? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). +
+
+ +- Do you still use `Archives`? +- Do you use the custom `generate_schema_name` macro? +- Do you use the `—non-destructive` flag? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). +
+
+
+#### Testing your changes before upgrading
+Once you have an idea about what code changes you'll need to make, you can start implementing them. We recommend that you create a separate dbt project, **Upgrade Project**, to test your changes before making them live in your main dbt project. In your **Upgrade Project**, connect to the same repository that you use for your main dbt project, but this time, set the development environment [settings](docs/dbt-cloud/cloud-configuring-dbt-cloud/cloud-choosing-a-dbt-version) to run the latest version of dbt Core. Next check out a branch `dbt-version-upgrade`, make the appropriate updates to your project (if needed), and see if your dbt project compiles and runs with the new version of dbt in the IDE. If jumping directly to the latest version of dbt is too far of a leap for your project, try iteratively getting your project to work on each successive minor version. There are years of development and a handful of breaking changes between two distant versions of dbt (e.g. 0.14 --> 1.0). There are far fewer between two subsequent versions of dbt, which is why upgrading regularly is important.
+
+Once you have your project compiling and running on the latest version of dbt in the development environment for your `dbt-version-upgrade` branch, try replicating one of your production jobs to run off your branch's code. You can do this by creating a new deployment environment for testing, setting the custom branch to 'ON' and referencing your `dbt-version-upgrade` branch. You'll also need to set the dbt version in this environment to the latest dbt Core version.
+
+
+
+Upgrading to v0.14.latest from v0.13
++ +- Do you still use `Archives`? +- Do you use the custom `generate_schema_name` macro? +- Do you use the `—non-destructive` flag? + +If you believe your project might be affected, read more details in the migration guide [here](/guides/migration/versions). +
@@ -50,5 +50,5 @@ The following diagram shows the network architecture for the hosted _Multi Tenan
- **EC2**: The hosted dbt Cloud deployments leverage a cluster of [AWS EC2](https://aws.amazon.com/ec2/) worker nodes to run the dbt Cloud application.
- **EBS**: In order to store application data, dbt Cloud leverages [AWS Elastic Block Store](https://aws.amazon.com/ebs/) mounted to the EC2 instances described above.
- **EFS**: An [AWS Elastic File System](https://aws.amazon.com/efs/) is provisioned for hosted deployments to store and manage local files from the dbt Cloud IDE.
-- **S3**: [AWS Simple Storage Service (S3)](https://aws.amazon.com/s3/) is used to store dbt Cloud application logs and artifacts (such as those generated from dbt job runs).
+- **S3**: [AWS Simple Storage Service (S3)](https://aws.amazon.com/s3/) is used to store dbt Cloud application logs and artifacts (such as those generated from dbt job runs).
- **RDS**: The hosted dbt Cloud application leverages [AWS Postgres RDS](https://aws.amazon.com/rds/postgresql/) to store application information such as accounts, users, environments, etc. Note that as explained in the [Data Warehouse Interaction](#data-warehouse-interaction) section above, no data from an associated warehouse is ever stored in this database.
diff --git a/website/docs/docs/deploy/cloud-ci-job.md b/website/docs/docs/deploy/cloud-ci-job.md
new file mode 100644
index 00000000000..8b4afe0a544
--- /dev/null
+++ b/website/docs/docs/deploy/cloud-ci-job.md
@@ -0,0 +1,202 @@
+---
+title: "dbt Cloud CI job"
+id: "cloud-ci-job"
+description: "You can enable continuous integration (CI) to test every single change prior to deploying the code to production just like in a software development workflow."
+---
+
+## Overview
+
+dbt Cloud makes it easy to test every single code change you make prior to deploying that new logic into production. Once you've connected your [GitHub account](/docs/collaborate/git/connect-github), [GitLab account](/docs/collaborate/git/connect-gitlab), or [Azure DevOps account](/docs/collaborate/git/connect-azure-devops), you can configure jobs to run when new pull requests are opened against your dbt repo.
+
+dbt Cloud will build the models affected by the new pull request code change in a temp schema, which acts as a quasi-staging environment, and will also run the tests that you've written for these models as a check. When the continuous integration (CI) job completes, the run status will be shown directly in the pull request. This makes it possible to deploy new code to production with confidence.
+
+:::info Draft Pull Requests
+
+Jobs will _not_ be triggered by draft pull requests. If you would like jobs to run on each new commit, please mark your pull request as **Ready for review**.
+
+:::
+
+:::info GitLab Compatibility
+
+GitLab Webhooks are available to only GitLab users who have a paid or self-hosted GitLab account.
+
+:::
+
+:::info Common Errors
+If you previously configured your dbt project by providing a generic git URL that clones using SSH, you need to [reconfigure the project](/docs/deploy/cloud-ci-job#reconnecting-your-dbt-project-to-use-dbt-clouds-native-integration-with-github-gitlab-or-azure-devops) to connect through dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps instead.
+:::
+
+## Understanding dbt Cloud Slim CI
+When a [dbt Cloud CI job is set up](/docs/deploy/cloud-ci-job#configuring-a-dbt-cloud-ci-job), dbt Cloud will listen for webhooks from GitHub, GitLab, or Azure DevOps indicating that a new PR has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the CI job. Crucially, this run will build into a temporary schema using the prefix `dbt_cloud_pr_`. This schema isolation acts as a quasi-staging environment, so that you can see the builds resulting from the code associated with the PR's commit sha. The unique schema name can be found in the run details for the given run, as shown below.
+
+
+
+You used dbt Cloud environment variables in your connection settings page
+
+
+To resolve this, remove environment variables in your connections settings.
+
+
+You have an empty/blank default schema
+
+
+To change this, edit and fill in your default schema.
+
+ You have overridden the
+
+You have overridden the generate_schema_name macro
+
+
+To resolve this, change your macro so that the temporary PR schema name contains the default prefix and review the guidance below:
+
+ • ✅ Temporary PR schema name contains the prefix
+ • ❌ Temporary PR schema name doesn't contain the prefix
+
+ + • ✅ Temporary PR schema name contains the prefix
dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing) + • ❌ Temporary PR schema name doesn't contain the prefix
dbt_cloud_pr_ (like marketing). +
+ You have overridden the
+
+
+
+Make the necessary changes to your project and double-check if the temporary PR schemas drop after a merge or close of the PR.
+
+## Troubleshooting
+
+### Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps
+
+If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud Slim CI.
+
+First, make sure you have the [native GitHub authentication](/docs/collaborate/git/connect-github), [native GitLab authentication](/docs/collaborate/git/connect-gitlab), or [native Azure DevOps authentication](/docs/collaborate/git/connect-azure-devops) set up depending on which git provider you use. After you have gone through those steps, head to **Account Settings**, select **Projects** and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.
+
+Once you're in the repository page, click **Edit** and then click **Disconnect Repository** at the bottom.
+You have overridden the generate_database_name macro
+
+
+If you assume that the project's default connection is to a database named
+ • ✅ Database remains the same as the connection default (like
+ • ❌ Database has changed from the default connection (like
+
+ analytics, review the guidance below to resolve this:
+ + • ✅ Database remains the same as the connection default (like
analytics) + • ❌ Database has changed from the default connection (like
dev). +
54.81.134.249
52.22.161.231 | ✅ | ✅ | ✅ | +| EMEA | eu-central-1 (Frankfurt) | emea.dbt.com | 3.123.45.39
3.126.140.248
3.72.153.148 | ❌ | ❌ | ✅ | +| Virtual Private dbt | Customized | Customized | Ask [Support](/guides/legacy/getting-help#dbt-cloud-support) for your IPs | ❌ | ❌ | ✅ | + + + diff --git a/website/docs/docs/dbt-cloud/deployments/single-tenant.md b/website/docs/docs/deploy/single-tenant.md similarity index 82% rename from website/docs/docs/dbt-cloud/deployments/single-tenant.md rename to website/docs/docs/deploy/single-tenant.md index 92945c50eaf..54cd764dadf 100644 --- a/website/docs/docs/dbt-cloud/deployments/single-tenant.md +++ b/website/docs/docs/deploy/single-tenant.md @@ -1,6 +1,6 @@ --- -id: single-tenant-deployment -title: Single Tenant +title: Single tenant +id: single-tenant --- The Single Tenant deployment environment provides a hosted alternative to the Multi Tenant (SaaS) dbt Cloud environment. While still managed and maintained by dbt Labs, the Single Tenant environment provides dedicated infrastructure with one or more instances of dbt Cloud that can only be accessed by a single customer. This is accomplished by spinning up all the necessary infrastructure with a re-usable Infrastructure as Code (IaC) deployment built with [Terraform](https://www.terraform.io/). The Single Tenant infrastructure lives in a dedicated AWS account and can be customized with certain configurations such as Firewall rules to limit ingress traffic or hosting in a specific AWS Region. @@ -13,5 +13,5 @@ _To learn more about setting up a dbt Cloud Single Tenant deployment, [please co For more information about the dbt Cloud Single Tenant deployment see the below. -- [Application Data Flows](/docs/dbt-cloud/deployments/deployment-architecture#application-data-flows) -- [Hosted Network Architecture](/docs/dbt-cloud/deployments/deployment-architecture#hosted-network-architecture) +- [Application Data Flows](/docs/deploy/architecture#application-data-flows) +- [Hosted Network Architecture](/docs/deploy/architecture#hosted-network-architecture) diff --git a/website/docs/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness.md b/website/docs/docs/deploy/source-freshness.md similarity index 80% rename from website/docs/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness.md rename to website/docs/docs/deploy/source-freshness.md index ca0073d30c4..5056e258eda 100644 --- a/website/docs/docs/dbt-cloud/using-dbt-cloud/cloud-snapshotting-source-freshness.md +++ b/website/docs/docs/deploy/source-freshness.md @@ -1,22 +1,23 @@ --- title: "Source freshness" -id: "cloud-snapshotting-source-freshness" +id: "source-freshness" description: "Validate that data freshness meets expectations and alert if stale." --- ## Data Source Freshness -dbt Cloud provides a helpful interface around dbt's [source data freshness](using-sources#snapshotting-source-data-freshness) calculations. When a dbt Cloud job is configured to snapshot source data freshness, dbt Cloud will render a user interface showing you the state of the most recent snapshot. This interface is intended to help you determine if your source data freshness is meeting the SLAs that you've defined for your organization. +dbt Cloud provides a helpful interface around dbt's [source data freshness](/docs/build/sources#snapshotting-source-data-freshness) calculations. When a dbt Cloud job is configured to snapshot source data freshness, dbt Cloud will render a user interface showing you the state of the most recent snapshot. This interface is intended to help you determine if your source data freshness is meeting the SLAs that you've defined for your organization. -
- Unsaved **(•)** — The IDE detects unsaved changes to your file/folder
- Modification **(M)** — The IDE detects a modification of existing files/folders
- Untracked **(U)** — The IDE detects changes made to new files or renamed files +| **Build, test, and run code** | Build, test, and run your project with a button click or by using the Cloud IDE command bar. +| **Drag and drop** | Drag and drop files located in the file explorer, and use the file breadcrumb on the top of the IDE for quick, linear navigation. Access adjacent files in the same file by right clicking on the breadcrumb file. +| **Organize tabs** | You can:
- Move your tabs around to reorganize your work in the IDE
- Right-click on a tab to view and select a list of actions to take
- Close multiple, unsaved tabs to batch save your work +| **Multiple selections** | You can make multiple selections for small and simultaneous edits. The below commands are a common way to add more cursors and allow you to insert cursors below or above with ease.
- Option-Command-Down arrow
- Option-Command-Up arrow
- Press Option and click on an area +| **Formatting** | Format your files with a click of a button, powered by [sqlfmt](http://sqlfmt.com/). +| **Git diff view** | Ability to see what has been changed in a file before you make a pull request. +| **dbt autocomplete** | There are four new types of autocomplete features to help you develop faster:
- Use `ref` to autocomplete your model names
- Use `source` to autocomplete your source name + table name
- Use `macro` to autocomplete your arguments
- Use `env var` to autocomplete env var +| **Dark mode** | Use dark mode in the Cloud IDE for a great viewing experience in low-light environments. + + +## Related docs + +- [dbt Cloud tips](/docs/get-started/dbt-cloud-tips) +- [Develop in the Cloud](docs/get-started/develop-in-the-cloud) +- [Guides](/docs/get-started/getting-started/overview) diff --git a/website/docs/docs/get-started/dbt-cloud-tips.md b/website/docs/docs/get-started/dbt-cloud-tips.md new file mode 100644 index 00000000000..687c6c97643 --- /dev/null +++ b/website/docs/docs/get-started/dbt-cloud-tips.md @@ -0,0 +1,95 @@ +--- +title: "dbt Cloud tips" +id: dbt-cloud-tips +description: "Check out any dbt Cloud and IDE-related tips." +sidebar_label: "dbt Cloud tips" +--- + +# dbt Cloud tips + +The Cloud IDE provides keyboard shortcuts, features, and development tips to help you work faster and be more productive. Use this Cloud IDE cheat sheet to help you quickly reference some common operations. + +## Most popular + +**Organize tabs** + +You can move your tabs around to reorganize your work in the IDE. You can also right-click on a tab to close multiple tabs or view and select a list of actions to take. Right-click in the editor to access other options, including text wrap. + +**Drag and drop** + +You can also drag and drop files in the file tree or in the editor. Use the file breadcrumb on the top of the IDE for quicker navigation. You can access adjacent files in the same file by right-clicking on the breadcrumb file. + +**Search across files** + +You can quickly search over all files in the IDE on your current project. To search, open the search bar by pressing Command-O to find text across all files in your current project and write your filename. You can view the results under the search text, which are grouped into files containing the match. You can click on the results to view them in the IDE. + +**Command bar and status** + +You can run commands from the command bar at the bottom of the IDE or by using the **Build** button. Use the [rich model selection syntax](/docs/reference/node-selection/syntax) to run [dbt commands](/docs/reference/dbt-commands) directly within dbt Cloud. You can also view the history, status, and logs of previous runs by clicking **Runs**. + +The status icon on the lower right corner of the IDE gives you an indicator of the health of your project. You can identify errors by clicking on the status icon for more details or by clicking **Restart the IDE**. + +**Find and replace** + +Press Command-F or Control-F to open the find-and-replace bar in the upper right corner of the current file in the IDE. The IDE highlights your search results in the current file and code outline. You can use the up and down arrows to see the match highlighted in the current file when there are multiple matches. To replace the text with something else, use the left arrow. + +**DAG in the IDE** + +Double-click a node in the directed acyclic graph (DAG) to open that file in a new tab. Click **Expand** on the upper right side of the DAG and use node selection syntax (`select` or `exclude`) to view a subset of your DAG. + +## IDE Keyboard shortcuts + +There are default keyboard shortcuts that can help make development more productive and easier for everyone. + +- Command-O or Control-O to select a file to open +- Command-P or Control-P to see command palette +- Hold Option-click-on-an-area to select multiple lines and perform a multi-edit. You can also press Command-E to perform this operation on the command line. +- Press Fn-F1 to view a list of the other editor shortcuts +- Command-Enter or Control-Enter to Preview your code +- Command-Shift-Enter or Control-Shift-Enter to Compile +- Highlight a portion of code and use the above shortcuts to Preview or Compile code +- Enter two underscores (__) in the IDE to reveal a list of dbt functions + +## Multiple selections + +You can make multiple selections for small and simultaneous edits. The below commands are a common way to add more cursors and allow you to insert cursors below or above with ease. + +- Option-Command-Down arrow +- Option-Command-Up arrow +- Press Option and click on an area + +## Package tips + +- Use the [dbt_codegen](https://hub.getdbt.com/dbt-labs/codegen/latest/) package to help you generate YML files for your models and sources and SQL files for your staging models. +- The [dbt_utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) package contains macros useful for daily development. For example, `date_spine` generates a table with all dates between the ones provided as parameters. +- The [dbt_project_evaluator](https://hub.getdbt.com/dbt-labs/dbt_project_evaluator/latest) package compares your dbt project against a list of our best practices and provides suggestions and guidelines on how to update your models. +- The [dbt_expectations](https://hub.getdbt.com/calogica/dbt_expectations/latest) package contains many tests beyond those built into dbt Core. +- The [dbt_audit_helper](https://hub.getdbt.com/#:~:text=adwords-,audit_helper,-codegen) package lets you compare the output of 2 queries. Use it when refactoring existing logic to ensure that the new results are identical. +- The [dbt_artifacts](https://hub.getdbt.com/brooklyn-data/dbt_artifacts/latest) package saves information about your dbt runs directly to your data platform so that you can track the performance of models over time. +- The [dbt_meta_testing](https://hub.getdbt.com/tnightengale/dbt_meta_testing/latest) package checks that your dbt project is sufficiently tested and documented. + +## Advanced tips + +- Use your folder structure as your primary selector method. `dbt build --select marts.marketing` is simpler and more resilient than relying on tagging every model. +- Think about jobs in terms of build cadences and SLAs. Run models that have hourly, daily, or weekly build cadences together. +- Use the [where config](/docs/reference/resource-configs/where) for tests to test an assertion on a subset of records. +- [store_failures](/docs/reference/resource-configs/store_failures) lets you examine records that cause tests to fail, so you can either repair the data or change the test as needed. +- Use [severity](/docs/reference/resource-configs/severity) thresholds to set an acceptable number of failures for a test. +- Use [incremental_strategy](/docs/build/incremental-models#about-incremental_strategy) in your incremental model config to implement the most effective behavior depending on the volume of your data and reliability of your unique keys. +- Set `vars` in your `dbt_project.yml` to define global defaults for certain conditions, which you can then override using the `--vars` flag in your commands. +- Use [for loops](/docs/get-started/learning-more/using-jinja#use-a-for-loop-in-models-for-repeated-sql) in Jinja to [DRY](https://docs.getdbt.com/terms/dry) up repetitive logic, such as selecting a series of columns that all require the same transformations and naming patterns to be applied. +- Instead of relying on post-hooks, use the [grants config](/docs/reference/resource-configs/grants) to apply permission grants in the warehouse resiliently. +- Define [source-freshness](/docs/build/sources#snapshotting-source-data-freshness) thresholds on your sources to avoid running transformations on data that has already been processed. +- Use the `+` operator on the left of a model `dbt build --select +model_name` to run a model and all of its upstream dependencies. Use the `+` operator on the right of the model `dbt build --select model_name+` to run a model and everything downstream that depends on it. +- Use `dir_name` to run all models in a package or directory. +- Use the `@` operator on the left of a model in a non-state-aware CI setup to test it. This operator runs all of a selection’s parents and children, and also runs the parents of its children, which in a fresh CI schema will likely not exist yet. +- Use the [--exclude flag](/docs/reference/node-selection/exclude) to remove a subset of models out of a selection. +- Use [state and deferral](/docs/deploy/cloud-ci-job#deferral-and-state-comparison) to create a slim CI setup. +- Use the [--full-refresh](/docs/reference/commands/run#refresh-incremental-models) flag to rebuild an incremental model from scratch. +- Use [seeds](/docs/build/seeds) to create manual lookup tables, like zip codes to states or marketing UTMs to campaigns. `dbt seed` will build these from CSVs into your warehouse and make them `ref` able in your models. +- Use [target.name](/docs/build/custom-schemas#an-alternative-pattern-for-generating-schema-names) to pivot logic based on what environment you’re using. For example, to build into a single development schema while developing, but use multiple schemas in production. + +## Related docs + +- [Getting started](/docs/get-started/getting-started/overview) +- [Develop in the Cloud](/docs/get-started/develop-in-the-cloud) diff --git a/website/docs/docs/get-started/develop-in-the-cloud.md b/website/docs/docs/get-started/develop-in-the-cloud.md new file mode 100644 index 00000000000..a29e18082ed --- /dev/null +++ b/website/docs/docs/get-started/develop-in-the-cloud.md @@ -0,0 +1,184 @@ +--- +title: "Develop in the Cloud" +id: develop-in-the-cloud +description: "Develop, test, run, and build in the Cloud IDE." +sidebar_label: Develop in the Cloud +--- + +
| Start up process | Work retention |
|---|---|
| + +There are three start-up states when using or launching the Cloud IDE: + +- Creation start — This is the state where you are starting the IDE for the first time. You can also view this as a *cold start* (see below), and you can expect this state to take longer because the git repository is being cloned. +- Cold start — This is the process of starting a new develop session, which will be available for you for three hours. The environment automatically turns off three hours after the last activity with the rpc server. This includes compile, preview, or any dbt invocation, however, it *does not* include editing and saving a file. +- Hot start — This is the state of resuming an existing or active develop session within three hours of the last activity. + + + | + + +The Cloud IDE needs explicit action to save your changes. There are three ways your work is stored: + +- Unsaved, local code — Any code you write is automatically available from your browser’s storage. You can see your changes but will lose them if you switch branches or browsers (another device or browser). +- Saved but uncommitted code — When you save a file, the data gets stored in your local storage (EFS storage). If you switch branches but don’t *commit* your saved changes, you will lose your changes. +- Committed code — This is stored in the branch with your git provider and you can check out other (remote) branches. + + + |
Compile
Build | This is where you can preview, compile or build your dbt project, as well as see the results and the DAG. | +| 7. | Lineage tab | You can see how models are used as building blocks from left to right to transform your data from raw sources into cleaned-up modular derived pieces and final outputs on the far right of the DAG. You can access files in the **Lineage** tab by double-clicking on a particular model. Expand the DAG into fullscreen to view the DAG view differently. Note: The default view is `+model+`, however, you can change it to `2+model+2`. | +| 8. | Change branches and view documentation | Change branches in fewer clicks and focus on developing. You can generate and view your [documentation](/docs/collaborate/build-and-view-your-docs) for your dbt project in real time. You can inspect and verify what your project's documentation will look like before you deploy your changes to production.| +| 9. | File state indicators | The file state indicators will indicate and track any action or changed made in your project. The indicators **M, U, and •** appear to the right of your file or folder name, and also under the **Changes** section. | +| 10. | Format button | This is where you can format your dbt project code. The new **Format** button formats your file and is powered by [sqlfmt](http://sqlfmt.com/).| + +## Build, compile, and run projects + +You can *build*, *compile*, *run* *, and test* dbt projects by using the command bar. The Cloud IDE will update in real time when you run models, tests, seeds, and operations. + +If a model or test fails, you can review the logs to find and fix the issue. + +You can also use dbt's [rich model selection syntax](/reference/node-selection/syntax) to [run dbt commands](/reference/dbt-commands) directly within dbt Cloud. + +
+
+Is there a cost to using the Cloud IDE?
+
+
+Not at all! You can use dbt Cloud when you sign up for the Free Developer plan, which comes with one developer seat. If you’d like to access more features or have more developer seats, you can upgrade your account to the Team or Enterprise plan. See dbt pricing plans for more details.
+
+
+Can I be a contributor to dbt Cloud?
+
+
+Anyone can contribute to the dbt project. And whether it's a dbt package, a plugin, dbt-core, or this documentation site, contributing to the open source code that supports the dbt ecosystem is a great way to level yourself up as a developer, and give back to the community. See Contributing for details on what to expect when contributing to the dbt open source software (OSS).
+
+
+What is the difference between developing on the Cloud IDE and on the CLI?
+
+
+There are two main ways to develop with dbt: using the web-based IDE in dbt Cloud or using the command-line interface (CLI) in dbt Core:
+ — + dbt Cloud IDE dbt Cloud is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project.
+ — + dbt Core CLI The command line interface (CLI) uses dbt Core, an open-source software that’s freely available. You can build your dbt project in a code editor, like Jetbrains or VSCode, and run dbt commands from the command line. +
+ + — + dbt Cloud IDE dbt Cloud is a web-based application that allows you to develop dbt projects with the IDE, includes a purpose-built scheduler, and provides an easier way to share your dbt documentation with your team. The IDE is a faster and more reliable way to deploy your dbt models and provides a real-time editing and execution environment for your dbt project.
+ — + dbt Core CLI The command line interface (CLI) uses dbt Core, an open-source software that’s freely available. You can build your dbt project in a code editor, like Jetbrains or VSCode, and run dbt commands from the command line. +
+
diff --git a/website/docs/docs/get-started/docker-install.md b/website/docs/docs/get-started/docker-install.md
new file mode 100644
index 00000000000..0a34cf4ec01
--- /dev/null
+++ b/website/docs/docs/get-started/docker-install.md
@@ -0,0 +1,54 @@
+---
+title: "Install with Docker"
+description: "You can use Docker to install dbt and adapter plugins from the command line."
+---
+
+dbt Core and all adapter plugins maintained by dbt Labs are available as [Docker](https://docs.docker.com/) images, and distributed via [GitHub Packages](https://docs.github.com/en/packages/learn-github-packages/introduction-to-github-packages).
+
+Using a prebuilt Docker image to install dbt Core in production has a few benefits: it already includes dbt-core, one or more database adapters, and pinned versions of all their dependencies. By contrast, `pip install dbt-core dbt-What type of support is provided with dbt Cloud?
+
+
+The global dbt Support team is available to help dbt Cloud users by email or in-product live chat. Developer and Team accounts offer 24x5 support, while Enterprise customers have priority access and options for custom coverage.
If you have project-related or modeling questions, review our Support page or dbt Community Slack to get help as well.
+ If you have project-related or modeling questions, review our Support page or dbt Community Slack to get help as well.
@@ -126,11 +126,11 @@ When developing locally, dbt connects to your data warehouse using a [profile](/
#### FAQs
-data warehouses .
1. Go to the [BigQuery credential wizard](https://console.cloud.google.com/apis/credentials/wizard). Make sure your new project is selected in the header. If you do not see your account or project, click your profile picture to the right and verify your are using the correct email account.
2. Select **+ Create Credentials** then select **Service account**.
@@ -100,7 +101,7 @@ In order to let dbt connect to your warehouse, you'll need to generate a keyfile
8. Select **Keys**.
9. Click **Add Key** then select **Create new key**.
10. Select **JSON** as the key type then click **Create**.
-11. You should be prompted to download the JSON file. Save it locally to an easy-to-remember spot, with a clear filename. For example, `dbt-user-creds.json`.
+11. You should be prompted to download the
@@ -198,34 +200,43 @@ Congratulations! At this point, you have created a Databricks account, loaded tr
### Option 2: Connect dbt Cloud and Databricks manually
-1. To manually setup dbt Cloud, you will need the SQL Endpoint connection information and to generate a user token. You can find your SQL endpoint connection information by going to the `Databricks UI > SQL > SQL Endpoints > Starter Endpoint > Connection details`. Save this information because you will need it later.
+#### Get endpoint and token information
- `. This is the schema connected directly to your development environment, and it's where your models will be built when running dbt within the Cloud IDE.
- - `User` and `token` that you saved in a previous step.
- - For the schema field, choose a development schema (this will be your default development database to build objects into). We recommend something in the form of dbt_{{ first initial, last name}} like `dbt_achen`.
-
-7. Then scroll to the top of the page to test your connection. Once successful, click `Continue`.
+5. Click **Test Connection** at the bottom. This verifies that dbt Cloud can access your Databricks workspace.
+6. If the connection test succeeds, click **Next**. If it fails, you may need to check your Databricks settings and credentials.
## Initialize your repository and start development
diff --git a/website/docs/tutorial/getting-set-up/setting-up-redshift.md b/website/docs/docs/get-started/getting-started/getting-set-up/setting-up-redshift.md
similarity index 82%
rename from website/docs/tutorial/getting-set-up/setting-up-redshift.md
rename to website/docs/docs/get-started/getting-started/getting-set-up/setting-up-redshift.md
index 7304910b98a..85300aca8ce 100644
--- a/website/docs/tutorial/getting-set-up/setting-up-redshift.md
+++ b/website/docs/docs/get-started/getting-started/getting-set-up/setting-up-redshift.md
@@ -3,6 +3,8 @@ title: "Set up and connect Redshift"
id: setting-up-redshift
description: "Set up Redshift with sample data and connect to dbt Cloud."
sidebar_label: "Set up and connect Redshift"
+pagination_prev: docs/get-started/getting-started/set-up-dbt-cloud
+pagination_next: docs/get-started/getting-started/building-your-first-project/build-your-first-models
---
## Introduction
@@ -32,7 +34,7 @@ Let’s get started by accessing your AWS account and setting up Redshift.
` naming convention.
+ - Schema: You’ll notice that the schema name has been auto created for you. By convention, this is `dbt_`. This is the schema connected directly to your development environment, and it's where your models will be built when running dbt within the Cloud IDE.
+
* **Role:** Leave blank for now. You can update this to a default Snowflake role in the future. * **Database:** `analytics`. This tells dbt to create new models in the analytics database. * **Warehouse:** `transforming`. This tells dbt to use the transforming warehouse we created earlier. -
`. This is the schema connected directly to your development environment, and it's where your models will be built when running dbt within the Cloud IDE.
* **Target name:** leave as default
- * **Threads:** Leave as 4. This is the number of simultaneous connects that dbt Cloud will make to build models concurently.
-
+ * **Threads:** Leave as 4. This is the number of simultaneous connects that dbt Cloud will make to build models concurrently.
+
+
diff --git a/website/docs/dbt-cli/install/homebrew.md b/website/docs/docs/get-started/homebrew-install.md
similarity index 85%
rename from website/docs/dbt-cli/install/homebrew.md
rename to website/docs/docs/get-started/homebrew-install.md
index aeeecb8c174..7617ff6d03f 100644
--- a/website/docs/dbt-cli/install/homebrew.md
+++ b/website/docs/docs/get-started/homebrew-install.md
@@ -1,5 +1,5 @@
---
-title: "Use Homebrew to install dbt"
+title: "Install with Homebrew"
description: "You can use Homebrew to install dbt Core and adapter plugins from the command line."
---
@@ -16,7 +16,7 @@ If that sounds like you, great! Homebrew makes it significantly easier to instal
- Installation with Homebrew can take longer than installing with other methods, because `brew` takes care of more setup behind the scenes
- If you're using an M1 Mac, we recommend that you install dbt via Homebrew with [Rosetta](https://support.apple.com/en-us/HT211861). This is necessary for certain dependencies that are only supported on Intel processors.
-If you're someone who prefers to manage Python environments yourself, such as having multiple versions of Python to switch between, we recommend you install dbt Core via [`pip` instead](install/pip).
+If you're someone who prefers to manage Python environments yourself, such as having multiple versions of Python to switch between, we recommend you install dbt Core via [`pip` instead](/docs/get-started/pip-install).
### Installing with Homebrew
@@ -28,7 +28,7 @@ brew install git
brew tap dbt-labs/dbt
```
-Now you're ready to install dbt. Once you know [which adapter](available-adapters) you're using, you can install it as `dbt-`. For instance, if using Postgres:
+Now you're ready to install dbt. Once you know [which adapter](supported-data-platforms) you're using, you can install it as `dbt-`. For instance, if using Postgres:
```shell
brew install dbt-postgres
@@ -36,7 +36,7 @@ brew install dbt-postgres
Everywhere below that you see ``, replace it with the adapter name you're using.
-**Note**: If you're using an adapter that isn't available as a Homebrew formula, we recommend you use [pip](install/pip) instead.
+**Note**: If you're using an adapter that isn't available as a Homebrew formula, we recommend you use [pip](/docs/get-started/pip-install) instead.
### Upgrading dbt and your adapter
diff --git a/website/docs/docs/get-started/installation-overview.md b/website/docs/docs/get-started/installation-overview.md
new file mode 100644
index 00000000000..cfb03a6b2cc
--- /dev/null
+++ b/website/docs/docs/get-started/installation-overview.md
@@ -0,0 +1,17 @@
+---
+title: "Installation overview"
+id: "installation"
+description: "You can install dbt Core using a few different tested methods."
+---
+
+You can install dbt Core on the command line by using one of these recommended methods:
+
+- [Use Homebrew to install dbt](/docs/get-started/homebrew-install) (recommended for MacOS + most popular plugins)
+- [Use pip to install dbt](/docs/get-started/pip-install)
+- [Use a Docker image to install dbt](/docs/get-started/docker-install)
+- [Install dbt from source](/docs/get-started/source-install)
+
+
+## About dbt adapters
+
+dbt works with a number of different data platforms (databases, query engines, and other SQL-speaking technologies). It does this by using a dedicated _adapter_ for each. When you install dbt, you'll also want to install the specific adapter for your database. For more details, see [Supported Data Platforms](supported-data-platforms).
diff --git a/website/docs/tutorial/learning-more/refactoring-legacy-sql.md b/website/docs/docs/get-started/learning-more/refactoring-legacy-sql.md
similarity index 85%
rename from website/docs/tutorial/learning-more/refactoring-legacy-sql.md
rename to website/docs/docs/get-started/learning-more/refactoring-legacy-sql.md
index c347cb86a44..210eb10a58b 100644
--- a/website/docs/tutorial/learning-more/refactoring-legacy-sql.md
+++ b/website/docs/docs/get-started/learning-more/refactoring-legacy-sql.md
@@ -1,14 +1,14 @@
---
-title: Refactoring legacy SQL to dbt SQL
+title: Refactoring legacy SQL to dbt
id: refactoring-legacy-sql
-description: This tutorial walks through refactoring a long SQL query (perhaps from a stored procedure) into modular dbt data models.
+description: This guide walks through refactoring a long SQL query (perhaps from a stored procedure) into modular dbt data models.
---
-You may have already learned how to build dbt models from scratch.
+You may have already learned how to build dbt models from scratch.
-But in reality, you probably already have some queries or stored procedures that power analyses and dashboards, and now you’re wondering how to port those into dbt.
+But in reality, you probably already have some queries or stored procedures that power analyses and dashboards, and now you’re wondering how to port those into dbt.
-There are two parts to accomplish this: migration and refactoring. In this tutorial we’re going to learn a process to help us turn legacy SQL code into modular dbt models.
+There are two parts to accomplish this: migration and refactoring. In this guide we’re going to learn a process to help us turn legacy SQL code into modular dbt models.
When migrating and refactoring code, it’s of course important to stay organized. We'll do this is by following several steps (jump directly from the right sidebar):
@@ -21,8 +21,9 @@ When migrating and refactoring code, it’s of course important to stay organize
Let's get into it!
-> Note: this tutorial is excerpted from the new dbt Learn On-demand Course, "Refactoring from Procedural SQL to dbt" - if you're curious, pick up the [free refactoring course here](https://courses.getdbt.com/courses/refactoring-sql-for-modularity), which includes example and practice refactoring projects.
-
+:::info More resources.
+This guide is excerpted from the new dbt Learn On-demand Course, "Refactoring SQL for Modularity" - if you're curious, pick up the [free refactoring course here](https://courses.getdbt.com/courses/refactoring-sql-for-modularity), which includes example and practice refactoring projects. Or for a more in-depth look at migrating DDL and DML from stored procedures check out [this guide](/guides/migration/tools/migrating-from-stored-procedures/1-migrating-from-stored-procedures).
+:::
## Migrate your existing SQL code
```sql
@@ -151,7 +150,7 @@ group by 1
Getting whitespace control right is often a lot of trial and error! We recommend that you prioritize the readability of your model code over the readability of the compiled code, and only do this as an extra polish.
## Use a macro to return payment methods
-Here, we've hardcoded the list of payment methods in our model. We may need to access this list from another model. A good solution here is to use a [variable](using-variables), but for the purpose of this tutorial, we're going to instead use a macro!
+Here, we've hardcoded the list of payment methods in our model. We may need to access this list from another model. A good solution here is to use a [variable](/docs/build/project-variables), but for the purpose of this tutorial, we're going to instead use a macro!
[Macros](jinja-macros#macros) in Jinja are pieces of code that can be called multiple times – they are analogous to a function in Python, and are extremely useful if you find yourself repeating code across multiple models.
@@ -312,7 +311,7 @@ Macros let analysts bring software engineering principles to the SQL they write.
A number of useful dbt macros have already been written in the [dbt-utils package](https://github.com/dbt-labs/dbt-utils). For example, the [get_column_values](https://github.com/dbt-labs/dbt-utils#get_column_values-source) macro from dbt-utils could be used instead of the `get_column_values` macro we wrote ourselves (saving us a lot of time, but at least we learnt something along the way!).
-Install the [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) package in your project (docs [here](package-management)), and then update your model to use the macro from the package instead:
+Install the [dbt-utils](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) package in your project (docs [here](/docs/build/packages)), and then update your model to use the macro from the package instead:
diff --git a/website/docs/dbt-cli/install/pip.md b/website/docs/docs/get-started/pip-install.md
similarity index 67%
rename from website/docs/dbt-cli/install/pip.md
rename to website/docs/docs/get-started/pip-install.md
index f6e1d378e72..5075a3e5086 100644
--- a/website/docs/dbt-cli/install/pip.md
+++ b/website/docs/docs/get-started/pip-install.md
@@ -1,17 +1,17 @@
---
-title: "Use pip to install dbt"
+title: "Install with pip"
description: "You can use pip to install dbt Core and adapter plugins from the command line."
---
-You need to use `pip` to install dbt Core on Windows or Linux operating systems. You should use [Homebrew](install/homebrew) for installing dbt Core on a MacOS.
+You need to use `pip` to install dbt Core on Windows or Linux operating systems. You should use [Homebrew](/docs/get-started/homebrew-install) for installing dbt Core on a MacOS.
-You can install dbt Core and plugins using `pip` because they are Python modules distributed on [PyPi](https://pypi.org/project/dbt/). We recommend using virtual environments when installing with `pip`.
+You can install dbt Core and plugins using `pip` because they are Python modules distributed on [PyPI](https://pypi.org/project/dbt/). We recommend using virtual environments when installing with `pip`.
-`. For instance, if using Postgres:
+Once you know [which adapter](supported-data-platforms) you're using, you can install it as `dbt-`. For example, if using Postgres:
```shell
pip install dbt-postgres
@@ -42,7 +42,7 @@ pip install --upgrade dbt-
### Install dbt-core only
-If you're building a tool that integrates with dbt Core, you may want to install the core library alone, without a database adapter. Note that you won't be able to use dbt as a CLI tool. Also, be advised that dbt Core's [Python API is currently unstable and undocumented](dbt-api).
+If you're building a tool that integrates with dbt Core, you may want to install the core library alone, without a database adapter. Note that you won't be able to use dbt as a CLI tool.
```shell
pip install dbt-core
@@ -50,7 +50,7 @@ pip install dbt-core
### `pip install dbt`
-Note that, as of v1.0.0, `pip install dbt` is no longer supported and will raise an explicit error. Since v0.13, the PyPi package named `dbt` was a simple "pass-through" of `dbt-core` and the four original database adapter plugins. For v1, we formalized that split.
+Note that, as of v1.0.0, `pip install dbt` is no longer supported and will raise an explicit error. Since v0.13, the PyPI package named `dbt` was a simple "pass-through" of `dbt-core` and the four original database adapter plugins. For v1, we formalized that split.
If you have workflows or integrations that relied on installing the package named `dbt`, you can achieve the same behavior going forward by installing the same five packages that it used:
diff --git a/website/docs/docs/get-started/run-your-dbt-projects.md b/website/docs/docs/get-started/run-your-dbt-projects.md
new file mode 100644
index 00000000000..ca7dd0b1757
--- /dev/null
+++ b/website/docs/docs/get-started/run-your-dbt-projects.md
@@ -0,0 +1,24 @@
+---
+title: "Run your dbt projects"
+id: "run-your-dbt-projects"
+---
+You can run your dbt projects with [dbt Cloud](/docs/get-started/dbt-cloud-features) and [dbt Core](https://github.com/dbt-labs/dbt-core). dbt Cloud is a hosted application where you can develop directly from a web browser. dbt Core is an open source project where you can develop from the command line.
+
+Among other features, dbt Cloud provides a development environment to help you build, test, run, and [version control](/docs/collaborate/git-version-control) your project faster. It also includes an easier way to share your [dbt project's documentation](/docs/collaborate/build-and-view-your-docs) with your team. These development tasks are directly built into dbt Cloud for an _integrated development environment_ (IDE). Refer to [Develop in the Cloud](/docs/get-started/develop-in-the-cloud) for more details.
+
+With dbt Core, you can run your dbt projects from the command line. The command line interface (CLI) is available from your computer's terminal application such as Terminal and iTerm. When using the command line, you can run commands and do other work from the current working directory on your computer. Before running the dbt project from the command line, make sure you are working in your dbt project directory. Learning terminal commands such as `cd` (change directory), `ls` (list directory contents), and `pwd` (present working directory) can help you navigate the directory structure on your system.
+
+When running your project from dbt Core or dbt Cloud, the commands you commonly use are:
+
+- [dbt run](/reference/commands/run) — Runs the models you defined in your project
+- [dbt build](/reference/commands/build) — Builds and tests your selected resources such as models, seeds, snapshots, and tests
+- [dbt test](/reference/commands/test) — Executes the tests you defined for your project
+
+For information on all dbt commands and their arguments (flags), see the [dbt command reference](/reference/dbt-commands). If you want to list all dbt commands from the command line, run `dbt --help`. To list a dbt command’s specific arguments, run `dbt COMMAND_NAME --help` .
+
+## Related docs
+
+- [How we set up our computers for working on dbt projects](https://discourse.getdbt.com/t/how-we-set-up-our-computers-for-working-on-dbt-projects/243)
+- [Model selection syntax](/reference/node-selection/syntax)
+- [Cloud IDE features](/docs/get-started/dbt-cloud-features#ide-features)
+- [Does dbt offer extract and load functionality?](/faqs/Project/transformation-tool)
diff --git a/website/docs/dbt-cli/install/from-source.md b/website/docs/docs/get-started/source-install.md
similarity index 82%
rename from website/docs/dbt-cli/install/from-source.md
rename to website/docs/docs/get-started/source-install.md
index 782461dfb46..6714e88cd10 100644
--- a/website/docs/dbt-cli/install/from-source.md
+++ b/website/docs/docs/get-started/source-install.md
@@ -1,14 +1,13 @@
---
-title: "Install dbt from source"
-id: "from-source"
+title: "Install from source"
description: "You can install dbt Core from its GitHub code source."
---
-dbt Core and almost all of its adapter plugins are open source software. As such, the codebases are freely available to download and build from source. You might install form source if you want the latest code or want to install dbt from a specific commit. This might be helpful when you are contributing changes, or if you want to debug a past change.
+dbt Core and almost all of its adapter plugins are open source software. As such, the codebases are freely available to download and build from source. You might install from source if you want the latest code or want to install dbt from a specific commit. This might be helpful when you are contributing changes, or if you want to debug a past change.
-To download form source, you would clone the repositories from GitHub, making a local copy, and then install the local version using `pip`.
+To download from source, you would clone the repositories from GitHub, making a local copy, and then install the local version using `pip`.
-Downloading and building dbt Core will enable you to contribute to the project by fixing a bug or implementing a sought-after feature. For more details, read the [contributing guidelines](https://github.com/dbt-labs/dbt/blob/HEAD/CONTRIBUTING.md).
+Downloading and building dbt Core will enable you to contribute to the project by fixing a bug or implementing a sought-after feature. For more details, read the [contributing guidelines](https://github.com/dbt-labs/dbt-core/blob/HEAD/CONTRIBUTING.md).
### Installing dbt Core
@@ -36,6 +35,6 @@ You do _not_ need to install `dbt-core` before installing an adapter plugin -- t
To install in editable mode, such as while contributing, use `pip install -e .` instead.
-tables and views .
+## About dbt
-dbt does the `T` in views . See [Managing environments](managing-environments) for more information on picking target schema names.
+### dbt Cloud
+dbt Cloud is the fastest and most reliable way to deploy dbt. Develop, test, schedule, and investigate data models all in one web-based UI. Read more about [Getting started with dbt Cloud](/docs/get-started/getting-started/set-up-dbt-cloud) and [dbt Cloud features](/docs/get-started/dbt-cloud-features).
-### dbt Commands
-A command is an instruction to execute dbt which compiles and runs SQL against the database.
+### dbt Core
-When you issue a dbt command, such as `run`, dbt:
+dbt Core is an open-source tool that enables data teams to transform data using analytics engineering best practices. You can install and use dbt Core on the command line. Read more about [Getting started with dbt Core](/docs/get-started/getting-started-dbt-core).
-1. Determines the order to execute the models in your project in.
-2. Generates the
-
-```sql
-select
- orders.id,
- orders.status,
- sum(case when payments.payment_method = 'bank_transfer' then payments.amount else 0 end) as bank_transfer_amount,
- sum(case when payments.payment_method = 'credit_card' then payments.amount else 0 end) as credit_card_amount,
- sum(case when payments.payment_method = 'gift_card' then payments.amount else 0 end) as gift_card_amount,
- sum(amount) as total_amount
-
-from {{ ref('base_orders') }} as orders
-left join {{ ref('base_payments') }} as payments on payments.order_id = orders.id
-```
-
-
-
-When compiled to executable SQL, dbt will replace the model specified in the `ref` function with the relation name.
-
-Importantly, dbt also uses the `ref` function to determine the sequence in which to execute the models – in the above example, `base_orders` and `base_payments` need to be built prior to building the `orders` model.
-
-`. The installation will include `dbt-core` and any other required dependencies, which may include both other dependencies and even other adapter plugins. Read more about [installing dbt](/docs/get-started/installation).
+
+## Adapter Taxonomy
+
+### Verified by dbt Labs
+
+In order to provide a more consistent and reliable experience, dbt Labs has a rigorous process by which we verify adapter plugins. The process covers aspects of development, documentation, user experience, and maintenance. These adapters earn a **Verified** designation so that users can have a certain level of trust and expectation when they use them. To learn more, see [Verifying a new adapter](/guides/advanced/adapter-development/7-verifying-a-new-adapter).
+
+We also welcome and encourage adapter plugins from the dbt community (see the below [Contributing to a pre-existing adapter](#contributing-to-a-pre-existing-adapter)). Please be mindful that these community maintainers are intrepid volunteers who donate their time and effort — so be kind, understanding, and help out where you can!
+
+### Maintainers
+
+Who made and maintains an adapter is certainly relevant, but we recommend using an adapter's verification status to determine the quality and health of an adapter. So far there are three categories of maintainers:
+
+| Supported by | Maintained By |
+| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| dbt Labs | dbt Labs maintains a set of adapter plugins for some of the most common databases, warehouses, and platforms. As for why particular data platforms were chosen, see ["Why Verify an Adapter"](7-verifying-a-new-adapter#why-verify-an-adapter) |
+| Partner | These adapter plugins are built and maintained by the same people who build and maintain the complementary data technology. |
+| Community | These adapter plugins are contributed and maintained by members of the community. 🌱 |
+
+
+## Contributing to dbt-core adapters
+
+### Contributing to a pre-existing adapter
+
+Community-supported plugins are works in progress, and anyone is welcome to contribute by testing and writing code. If you're interested in contributing:
+
+- Join both the dedicated channel, [#adapter-ecosystem](https://getdbt.slack.com/archives/C030A0UF5LM), in [dbt Slack](https://community.getdbt.com/) and the channel for your adapter's data store (see **Slack Channel** column of above tables)
+- Check out the open issues in the plugin's source repository (follow relevant link in **Adapter Repository** column of above tables)
+
+### Creating a new adapter
+
+If you see something missing from the lists above, and you're interested in developing an integration, read more about adapters and how they're developed in the [Adapter Development](/guides/advanced/adapter-development/1-what-are-adapters) section.
+
+If you have a new adapter, please add it to this list using a pull request! See [Documenting your adapter](5-documenting-a-new-adapter) for more information.
+
+[^1]: Here are the two different adapters. Use the PyPI package name when installing with `pip`
+
+ | Adapter repo name | PyPI package name |
+ | ----------------- | -------------------- |
+ | `dbt-athena` | `dbt-athena-adapter` |
+ | `dbt-layer` | `dbt-layer-bigquery` |
diff --git a/website/docs/docs/use-dbt-semantic-layer/avail-sl-integrations.md b/website/docs/docs/use-dbt-semantic-layer/avail-sl-integrations.md
new file mode 100644
index 00000000000..9abd608300e
--- /dev/null
+++ b/website/docs/docs/use-dbt-semantic-layer/avail-sl-integrations.md
@@ -0,0 +1,28 @@
+---
+title: "Available integrations"
+id: avail-sl-integrations
+description: "Review a wide range of partners you can integrate and query with the dbt Semantic Layer."
+sidebar_label: "Available integrations"
+---
+
+# Available integrations
+
+A wide variety of data applications across the modern data stack natively integrate with the dbt Semantic Layer and dbt metrics — from Business Intelligence tools to notebooks, data catalogs, and more.
+
+There are two categories of partner integrations:
+
+- **dbt Semantic Layer partners** - Semantic Layer integrations are capable of interactively querying dbt metrics, surfacing the underlying data in partner tools. This refers to a group of dbt Labs partners who have full end-to-end integrations with the dbt Semantic Layer and leverage the dbt Server.
+
+- **Metrics Ready partners** - Metrics Ready integrations facilitate building, discovery, and collaborating on dbt metric definitions.
+
+For information on the different integration partners, their documentation, and more, review the [dbt Semantic Layer integrations](https://www.getdbt.com/product/semantic-layer-integrations) page.
+
+
+
+- Have a multi-tenant dbt Cloud account, hosted in North America
+- Have both your production and development environments running dbt version 1.3 or higher
+- Use Snowflake data platform
+- Install the dbt metrics package version
+- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+
+
+
+- Have a multi-tenant dbt Cloud account, hosted in North America
+- Have both your production and development environments running dbt version 1.2 (latest)
+- Use Snowflake data platform
+- Install the dbt metrics package version
+- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+
+
_* Available during Public Preview only_ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise) | +| **[Metadata API](/docs/dbt-cloud-apis/metadata-api)** | Accesses metric definitions primarily via integrations and is the source of truth for objects defined in dbt projects (like models, macros, sources, metrics). The Metadata API is updated at the end of every dbt Cloud run. | ❌ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise | + +
+ +- [How to design and structure dbt metrics: Recommendations for getting started](https://docs.getdbt.com/blog/how-to-design-and-structure-metrics) blog to understand best practices for designing and structuring metrics in your dbt project +- [dbt metrics](docs/build/metrics) for in-depth detail on attributes, filters, how to define and query your metrics and [dbt-metrics package](https://github.com/dbt-labs/dbt_metrics) +- [dbt Semantic Layer quickstart](/docs/use-dbt-semantic-layer/quickstart-semantic-layer) to get started +- [Understanding the components of the dbt Semantic Layer](https://docs.getdbt.com/blog/understanding-the-components-of-the-dbt-semantic-layer) blog post to see further examples + +Review our helpful metrics video below, which explains what metrics are, why they're important and how you can get started: + +
diff --git a/website/docs/docs/use-dbt-semantic-layer/quickstart-semantic-layer.md b/website/docs/docs/use-dbt-semantic-layer/quickstart-semantic-layer.md new file mode 100644 index 00000000000..2ed60e32ded --- /dev/null +++ b/website/docs/docs/use-dbt-semantic-layer/quickstart-semantic-layer.md @@ -0,0 +1,321 @@ +--- +title: "Quickstart" +id: quickstart-semantic-layer +description: "Define metrics and set up the dbt Semantic Layer" +sidebar_label: "Quickstart" +--- + +# dbt Semantic Layer quickstart + +
+
+- Have a multi-tenant dbt Cloud account, hosted in North America
+- Have both your production and development environments running dbt version 1.3 or higher
+- Use Snowflake data platform
+- Install the dbt metrics package version
+- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+
+
+
+- Have a multi-tenant dbt Cloud account, hosted in North America
+- Have both your production and development environments running dbt version 1.2 (latest)
+- Use Snowflake data platform
+- Install the dbt metrics package version
+- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+
+
+
+```yml
+packages:
+ - package: dbt-labs/metrics
+ version: [">=1.3.0", "<1.4.0"]
+```
+
+
+
+
+
+```yml
+packages:
+ - package: dbt-labs/metrics
+ version: [">=0.3.0", "<0.4.0"]
+```
+
+
+
+
+
+```yml
+packages:
+ - package: dbt-labs/metrics
+ version: [">=0.2.0", "<0.3.0"]
+```
+
+
+
+
+1. Paste the dbt metrics package code in your `packages.yml` file.
+2. Run the [`dbt deps` command](/reference/commands/deps) to install the package.
+3. If you see a successful result, you have now installed the dbt metrics package successfully!
+4. If you have any errors during the `dbt deps` command run, review the system logs for more information on how to resolve them. Make sure you use a dbt metrics package that’s compatible with your dbt environment version.
+
+
+
+```sql
+version: 2
+
+metrics:
+ - name: expenses
+ label: Expenses
+ model: ref('orders')
+ description: "The total expenses of our jaffle business"
+
+ calculation_method: sum
+ expression: amount / 4
+
+ timestamp: order_date
+ time_grains: [day, week, month, year]
+
+ dimensions:
+ - customer_status
+ - had_credit_card_payment
+ - had_coupon_payment
+ - had_bank_transfer_payment
+ - had_gift_card_payment
+
+ filters:
+ - field: status
+ operator: '='
+ value: "'completed'"
+```
+
+
+
+
+```sql
+version: 2
+
+metrics:
+ - name: expenses
+ label: Expenses
+ model: ref('orders')
+ description: "The total expenses of our jaffle business"
+
+ type: sum
+ sql: amount / 4
+
+ timestamp: order_date
+ time_grains: [day, week, month, year]
+
+ dimensions:
+ - customer_status
+ - had_credit_card_payment
+ - had_coupon_payment
+ - had_bank_transfer_payment
+ - had_gift_card_payment
+
+ filters:
+ - field: status
+ operator: '='
+ value: "'completed'"
+```
+
+
+1. Click **Save** and then **Compile** the code.
+2. Commit and merge the code changes that contain the metric definitions.
+3. If you'd like to further design and define your own metrics, review the following documentation:
+
+ - [dbt metrics](/docs/build/metrics) will povide you in-depth detail on attributes, properties, filters, and how to define and query metrics.
+
+ - Review [How to design and structure dbt metrics: Recommendations for getting started](https://docs.getdbt.com/blog/how-to-design-and-structure-metrics) blog to understand best practices for designing and structuring metrics in your dbt project.
+
+## Develop and query metrics
+
+You can dynamically develop and query metrics directly in dbt and verify their accuracy _before_ running a job in the deployment environment by using the `metrics.calculate` and `metrics.develop` macros.
+
+To understand when and how to use the macros above, review [dbt metrics](/docs/build/metrics) and make sure you install the [dbt_metrics package](https://github.com/dbt-labs/dbt_metrics) first before using the above macros.
+
+:::info 📌
+
+**Note:** You will need access to dbt Cloud and the dbt Semantic Layer from your integrated partner tool of choice.
+
+:::
+
+## Run your production job
+
+Once you’ve defined metrics in your dbt project, you can perform a job run in your deployment environment to materialize your metrics. The deployment environment is only supported for the dbt Semantic Layer at this moment.
+
+1. Go to **Deploy** in the navigation and select **Jobs** to re-run the job with the most recent code in the deployment environment.
+2. Your metric should appear as a red node in the dbt Cloud IDE and dbt directed acyclic graphs (DAG).
+
+
+ + +## Next steps + +Are you ready to define your own metrics and bring consistency to data consumers? Review the following documents to understand how to structure, define, and query metrics, and set up the dbt Semantic Layer: + +- [How to design and structure dbt metrics: Recommendations for getting started](https://docs.getdbt.com/blog/how-to-design-and-structure-metrics) to understand best practices for designing and structuring metrics in your dbt project +- [dbt metrics](/docs/build/metrics) for in-depth detail on attributes, properties, filters, and how to define and query metrics +- [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-semantic-layer) to learn about the dbt Semantic Layer +- [Understanding the components of the dbt Semantic Layer](https://docs.getdbt.com/blog/understanding-the-components-of-the-dbt-semantic-layer) blog post to see further examples +- [Integrated partner tools](https://www.getdbt.com/product/semantic-layer-integrations) for info on the different integration partners and their documentation +- [dbt Server repo](https://github.com/dbt-labs/dbt-server), which is a persisted HTTP server that wraps dbt core to handle RESTful API requests for dbt operations. diff --git a/website/docs/docs/use-dbt-semantic-layer/set-dbt-semantic-layer.md b/website/docs/docs/use-dbt-semantic-layer/set-dbt-semantic-layer.md new file mode 100644 index 00000000000..90fabea0bca --- /dev/null +++ b/website/docs/docs/use-dbt-semantic-layer/set-dbt-semantic-layer.md @@ -0,0 +1,54 @@ +--- +title: "Set up the dbt Semantic Layer" +id: setup-dbt-semantic-layer +description: "You can set up the dbt Semantic Layer in dbt Cloud." +sidebar_label: "Set up the dbt Semantic Layer" +--- + +With the dbt Semantic Layer, you'll be able to centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. Configure the dbt Semantic Layer in dbt Cloud to connect with your integrated partner tool. + +## Prerequisites + +Before you set up the dbt Semantic Layer, make sure you meet the following: + +
+
+- Have a multi-tenant dbt Cloud account, hosted in North America
+- Have both your production and development environments running dbt version 1.3 or higher
+- Use Snowflake data platform
+- Install the dbt metrics package version
+- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+
+
+
+- Have a multi-tenant dbt Cloud account, hosted in North America
+- Have both your production and development environments running dbt version 1.2 (latest)
+- Use Snowflake data platform
+- Install the dbt metrics package version
+- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+
+
+
+ + +## Related docs + +- [Integrated partner tools](https://www.getdbt.com/product/semantic-layer-integrations) for info on the different integration partners and their documentation +- [Product architecture](/docs/use-dbt-semantic-layer/dbt-semantic-layer#product-architecture) page for more information on plan availability +- [dbt metrics](/docs/build/metrics) for in-depth detail on attributes, properties, filters, and how to define and query metrics +- [dbt Server repo](https://github.com/dbt-labs/dbt-server), which is a persisted HTTP server that wraps dbt core to handle RESTful API requests for dbt operations diff --git a/website/docs/faqs/Accounts/change-billing.md b/website/docs/faqs/Accounts/change-billing.md new file mode 100644 index 00000000000..65e47cc9c4a --- /dev/null +++ b/website/docs/faqs/Accounts/change-billing.md @@ -0,0 +1,11 @@ +--- +title: How can I update my billing information? +description: "Instructions on how to update your billing info" +sidebar_label: 'How to update billing' +id: change-billing +--- + + +If you want to change your account's credit card details, select the gear menu in the upper right corner of dbt Cloud. Go to Account Settings → Billing → Payment Information. In the upper right corner of Payment Information, click **Edit** to enter the new credit card details. Only the _account owner_ can make this change. + +To change your billing name or location address, send our Support team a message at support@getdbt.com with the newly updated information, and we can make that change for you! diff --git a/website/docs/faqs/configurable-snapshot-path.md b/website/docs/faqs/Accounts/configurable-snapshot-path.md similarity index 76% rename from website/docs/faqs/configurable-snapshot-path.md rename to website/docs/faqs/Accounts/configurable-snapshot-path.md index 66f084ee775..45e51f20dbf 100644 --- a/website/docs/faqs/configurable-snapshot-path.md +++ b/website/docs/faqs/Accounts/configurable-snapshot-path.md @@ -1,5 +1,8 @@ --- title: Can I store my snapshots in a directory other than the `snapshot` directory in my project? +description: "You can see how to change snapshots directory in your project" +sidebar_label: 'Store snapshot in other directory' +id: configurable-snapshot-path --- By default, dbt expects your snapshot files to be located in the `snapshots` subdirectory of your project. diff --git a/website/docs/faqs/Accounts/dbt-specific-jinja.md b/website/docs/faqs/Accounts/dbt-specific-jinja.md new file mode 100644 index 00000000000..e2623491007 --- /dev/null +++ b/website/docs/faqs/Accounts/dbt-specific-jinja.md @@ -0,0 +1,8 @@ +--- +title: What parts of Jinja are dbt-specific? +description: "You can check which Jinja functions are dbt-specific" +sidebar_label: 'dbt-specific Jinja' +id: dbt-specific-jinja +--- + +There are certain expressions that are specific to dbt — these are documented in the [Jinja function reference](/reference/dbt-jinja-functions) section of these docs. Further, docs blocks, snapshots, andmaterializations are custom Jinja _blocks_ that exist only in dbt.
diff --git a/website/docs/faqs/Accounts/git-account-in-use.md b/website/docs/faqs/Accounts/git-account-in-use.md
new file mode 100644
index 00000000000..a84577e6ede
--- /dev/null
+++ b/website/docs/faqs/Accounts/git-account-in-use.md
@@ -0,0 +1,15 @@
+---
+title: Why am I getting an "account in use" error?
+description: "What to do when you receive a Git account in use error"
+sidebar_label: 'Git account in use'
+id: git-account-in-use
+---
+
+If you're receiving an 'Account in use' error when trying to integrate GitHub in your Profile page, this is because the Git integration is a 1-to-1 integration, so you can only have your Git account linked to one dbt Cloud user account.
+
+Here are some steps to take to get you unstuck:
+
+* Log in to the dbt Cloud account integrated with your Git account. Go to your user profile and click on Integrations to remove the link.
+
+If you don't remember which dbt Cloud account is integrated, please email dbt Support at support@getdbt.com and we'll do our best to disassociate the integration for you.
+
diff --git a/website/docs/faqs/Accounts/payment-accepted.md b/website/docs/faqs/Accounts/payment-accepted.md
new file mode 100644
index 00000000000..2e26063c684
--- /dev/null
+++ b/website/docs/faqs/Accounts/payment-accepted.md
@@ -0,0 +1,10 @@
+---
+title: Can I pay via invoice?
+description: "What payments types are accepted"
+sidebar_label: 'Can I pay invoice'
+id: payment-accepted
+---
+
+Presently for Team plans, self-service dbt Cloud payments must be made via credit card and by default, they will be billed monthly based on the number of active developer seats.
+
+We don't have any plans to do invoicing for self-service teams in the near future, but we *do* currently support invoices for companies on the **dbt Cloud Enterprise plan.** Feel free to [contact us](https://www.getdbt.com/contact) to build your Enterprise pricing.
diff --git a/website/docs/faqs/Accounts/slack.md b/website/docs/faqs/Accounts/slack.md
new file mode 100644
index 00000000000..01001141e2e
--- /dev/null
+++ b/website/docs/faqs/Accounts/slack.md
@@ -0,0 +1,8 @@
+---
+title: How do I set up Slack notifications?
+description: "Instructions on how to set up slack notifications"
+sidebar_label: 'How to set up Slack'
+id: slack
+---
+
+
diff --git a/website/docs/faqs/document-all-columns.md b/website/docs/faqs/Docs/document-all-columns.md
similarity index 71%
rename from website/docs/faqs/document-all-columns.md
rename to website/docs/faqs/Docs/document-all-columns.md
index 19fd5c4f96a..0de7881bfdd 100644
--- a/website/docs/faqs/document-all-columns.md
+++ b/website/docs/faqs/Docs/document-all-columns.md
@@ -1,5 +1,8 @@
---
title: Do I need to add a yaml entry for column for it to appear in the docs site?
+description: "All columns appear in your docs site"
+sidebar_label: 'Types of columns included in doc site'
+id: document-all-columns
---
Fortunately, no!
diff --git a/website/docs/faqs/document-other-resources.md b/website/docs/faqs/Docs/document-other-resources.md
similarity index 65%
rename from website/docs/faqs/document-other-resources.md
rename to website/docs/faqs/Docs/document-other-resources.md
index 58dda592d36..b5d612fbcfc 100644
--- a/website/docs/faqs/document-other-resources.md
+++ b/website/docs/faqs/Docs/document-other-resources.md
@@ -1,5 +1,8 @@
---
title: Can I document things other than models, like sources, seeds, and snapshots?
+description: "Using the description key to document resources"
+sidebar_label: 'Document other resources'
+id: document-other-resources
---
Yes! You can document almost everything in your project using the `description:` key. Check out the reference docs on [descriptions](description) for more info!
diff --git a/website/docs/faqs/documenting-macros.md b/website/docs/faqs/Docs/documenting-macros.md
similarity index 82%
rename from website/docs/faqs/documenting-macros.md
rename to website/docs/faqs/Docs/documenting-macros.md
index 7db45b3ef74..71211f2c100 100644
--- a/website/docs/faqs/documenting-macros.md
+++ b/website/docs/faqs/Docs/documenting-macros.md
@@ -1,12 +1,15 @@
---
title: How do I document macros?
+description: "You can use a schema file to document macros"
+sidebar_label: 'Document macros'
+id: documenting-macros
---
The `macros:` key is new in 0.16.0.
To document macros, use a [schema file](macro-properties) and nest the configurations under a `macros:` key
-## Example:
+## Example
diff --git a/website/docs/faqs/Docs/long-descriptions.md b/website/docs/faqs/Docs/long-descriptions.md
new file mode 100644
index 00000000000..f353050468e
--- /dev/null
+++ b/website/docs/faqs/Docs/long-descriptions.md
@@ -0,0 +1,23 @@
+---
+title: How do I write long-form explanations in my descriptions?
+description: "Write long descriptions in your documentation"
+sidebar_label: 'Write long descriptions'
+id: long-descriptions
+---
+If you need more than a sentence to explain a model, you can:
+
+1. Split your description over multiple lines ([yaml docs](https://yaml-multiline.info/)), like so:
+
+ ```yml
+ version: 2
+
+ models:
+ - name: customers
+ description: >
+ Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
+ tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
+ quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
+ consequat.
+ ```
+
+2. Use a [docs block](/documentation#using-docs-blocks) to write the description in a Markdown file.
diff --git a/website/docs/faqs/Docs/sharing-documentation.md b/website/docs/faqs/Docs/sharing-documentation.md
new file mode 100644
index 00000000000..4c6e0e84f77
--- /dev/null
+++ b/website/docs/faqs/Docs/sharing-documentation.md
@@ -0,0 +1,8 @@
+---
+title: How do I share my documentation with my team members?
+description: "Use read-only seats to share documentation"
+sidebar_label: 'Share documentation with teammates'
+id: sharing-documentation
+---
+
+If you're using dbt Cloud to deploy your project, and have the [Team plan](https://www.getdbt.com/pricing/), you can have up to 5 read-only users, who will be able access the documentation for your project.
diff --git a/website/docs/faqs/beta-release.md b/website/docs/faqs/Environments/beta-release.md
similarity index 75%
rename from website/docs/faqs/beta-release.md
rename to website/docs/faqs/Environments/beta-release.md
index 6cfea736780..5eef07d3510 100644
--- a/website/docs/faqs/beta-release.md
+++ b/website/docs/faqs/Environments/beta-release.md
@@ -1,4 +1,7 @@
---
title: What is a beta release?
+description: "How to try out beta features"
+sidebar_label: 'What is a beta release?'
+id: beta-release
---
This is a chance to try out brand-new functionality. You get to start planning for use cases that the next minor version will unlock. We get to hear from you about unexpected behavior and nasty bugs, so that the release candidate has more polish and fewer surprises.
diff --git a/website/docs/faqs/Environments/diff-database-environment.md b/website/docs/faqs/Environments/diff-database-environment.md
new file mode 100644
index 00000000000..49675540b80
--- /dev/null
+++ b/website/docs/faqs/Environments/diff-database-environment.md
@@ -0,0 +1,8 @@
+---
+title: Can I set a different connection at the environment level?
+description: "Separate projects for different environments workaround"
+sidebar_label: 'Set different database connections at environment level'
+id: diff-database-environment
+---
+
+Although you're unable to set a different connection at the environment level, there is a workaround where you can have separate projects for their different environments and link them to different hostnames, while still connecting to the same repo.
diff --git a/website/docs/faqs/profile-env-vars.md b/website/docs/faqs/Environments/profile-env-vars.md
similarity index 56%
rename from website/docs/faqs/profile-env-vars.md
rename to website/docs/faqs/Environments/profile-env-vars.md
index 32912361343..d9cea7946de 100644
--- a/website/docs/faqs/profile-env-vars.md
+++ b/website/docs/faqs/Environments/profile-env-vars.md
@@ -1,4 +1,7 @@
---
title: Can I use environment variables in my profile?
+description: "Use env_var in your profile"
+sidebar_label: 'Use env_var in your profile'
+id: profile-env-vars
---
Yes! Check out the docs on [environment variables](env_var) for more information.
diff --git a/website/docs/faqs/Environments/profile-name.md b/website/docs/faqs/Environments/profile-name.md
new file mode 100644
index 00000000000..9f9d8b0ae59
--- /dev/null
+++ b/website/docs/faqs/Environments/profile-name.md
@@ -0,0 +1,7 @@
+---
+title: What should I name my profile?
+description: "Use company name for profile name"
+sidebar_label: 'Naming your profile'
+id: profile-name
+---
+We typically use a company name for a profile name, and then use targets to differentiate between `dev` and `prod`. Check out the docs on [managing environments](/docs/collaborate/environments) for more information.
diff --git a/website/docs/faqs/Environments/target-names.md b/website/docs/faqs/Environments/target-names.md
new file mode 100644
index 00000000000..6d3cd78b8e0
--- /dev/null
+++ b/website/docs/faqs/Environments/target-names.md
@@ -0,0 +1,8 @@
+---
+title: What should I name my target?
+description: "Using targets to clarify development and production runs"
+sidebar_label: 'Naming your target'
+id: target-names
+---
+
+We typically use targets to differentiate between development and production runs of dbt, naming the targets `dev` and `prod` respectively. Check out the docs on [managing environments](/docs/collaborate/environments) for more information.
diff --git a/website/docs/faqs/Git/gitignore.md b/website/docs/faqs/Git/gitignore.md
new file mode 100644
index 00000000000..fb097bb4043
--- /dev/null
+++ b/website/docs/faqs/Git/gitignore.md
@@ -0,0 +1,25 @@
+---
+title: Why can't I checkout a branch or create a new branch?
+description: "Add or fill in gitignore file"
+sidebar_label: 'Unable to checkout or create branch'
+id: gitignore
+---
+
+If you're finding yourself unable to revert changes, check out a branch or click commit - this is usually do to your project missing a [.gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file OR your gitignore file doesn't contain the necessary content inside the folder.
+
+This is what causes that 'commit' git action button to display. No worries though - to fix this, you'll need to complete the following steps in order:
+
+1. In the Cloud IDE, add the missing .gitignore file or contents to your project. You'll want to make sure the .gitignore file includes the following:
+
+ ```shell
+ target/
+ dbt_modules/
+ dbt_packages/
+ logs/
+ ```
+
+2. Once you've added that, make sure to save and commit.
+
+3. Navigate to the same branch in your remote repository (which can be accessed directly through your git provider's web interface) and delete the logs, target, and dbt_modules/dbt_packages folders.
+
+4. Go back into the Cloud IDE and reclone your repository. This can be done by clicking on the green "ready" in the bottom right corner of the IDE (next to the command bar), and then clicking the orange "reclone repo" button in the pop up.
diff --git a/website/docs/faqs/Git/gitlab-authentication.md b/website/docs/faqs/Git/gitlab-authentication.md
new file mode 100644
index 00000000000..0debdf87873
--- /dev/null
+++ b/website/docs/faqs/Git/gitlab-authentication.md
@@ -0,0 +1,29 @@
+---
+title: I'm seeing a Gitlab authentication out of date error loop
+description: "GitLab and dbt Cloud deploy key mismatch "
+sidebar_label: 'GitLab authentication out of date'
+id: gitlab-authentication
+---
+
+If you're seeing a 'GitLab Authentication is out of date' 500 server error page - this usually occurs when the deploy key in the repository settings in both dbt Cloud and GitLab do not match.
+
+No worries - this is a current issue the dbt Labs team is working on and we have a few workarounds for you to try:
+
+### 1st Workaround
+
+1. Disconnect repo from project in dbt Cloud.
+2. Go to Gitlab and click on Settings > Repository.
+3. Under Repository Settings, remove/revoke active dbt Cloud deploy tokens and deploy keys.
+4. Attempt to reconnect your repository via dbt Cloud.
+5. You would then need to check Gitlab to make sure that the new deploy key is added.
+6. Once confirmed that it's added, refresh dbt Cloud and try developing once again.
+
+### 2nd Workaround
+
+1. Keep repo in project as is -- don't disconnect.
+2. Copy the deploy key generated in dbt Cloud.
+3. Go to Gitlab and click on Settings > Repository.
+4. Under Repository Settings, manually add to your Gitlab project deploy key repo (with `Grant write permissions` box checked).
+5. Go back to dbt Cloud, refresh your page and try developing again.
+
+If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Git/gitlab-selfhosted.md b/website/docs/faqs/Git/gitlab-selfhosted.md
new file mode 100644
index 00000000000..c02c363cf3a
--- /dev/null
+++ b/website/docs/faqs/Git/gitlab-selfhosted.md
@@ -0,0 +1,10 @@
+---
+title: Can self-hosted GitLab instances only be connected via dbt Cloud Enterprise plans?
+description: "Self hosted GitLab instances plan availability"
+sidebar_label: 'Self-hosted GitLab instances available for dbt Cloud Enterprise'
+id: gitlab-selfhosted
+---
+
+Presently yes, this is only available to Enterprise users. This is because of the way you have to set up the GitLab app redirect URL for auth, which can only be customized if you're a user on an Enterprise plan.
+
+Check out our [pricing page](https://www.getdbt.com/pricing/) for more information or feel free to [contact us](https://www.getdbt.com/contact) to build your Enterprise pricing.
diff --git a/website/docs/faqs/Git/google-cloud-repo.md b/website/docs/faqs/Git/google-cloud-repo.md
new file mode 100644
index 00000000000..1e88ac9a6ed
--- /dev/null
+++ b/website/docs/faqs/Git/google-cloud-repo.md
@@ -0,0 +1,18 @@
+---
+title: How can I connect dbt to a Google Source repository?
+description: "Use SSH URL method to connect to Google Source Repo"
+sidebar_label: 'Connect dbt to a Google Source repository'
+id: google-cloud-repo
+
+---
+
+Although we don't officially support Google Cloud as a git repository, the below workaround using the SSH URL method should help you to connect:
+
+- First: "import" your Repository into dbt Cloud using the SSH URL provided to you by GCP. That will look something like:
+`ssh://drew@fishtownanalytics.com@source.developers.google.com:2022/p/dbt-integration-tests/r/drew-debug`
+
+- After importing the repo, you should see a public key generated by dbt Cloud for the repository. You'll want to copy that public key into a new SSH Key for your user here: [https://source.cloud.google.com/user/ssh_keys](https://source.cloud.google.com/user/ssh_keys)
+
+- After saving this SSH key, dbt Cloud should be able to read and write to this repo.
+
+If you've tried the workaround above and are still having issues connecting - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Git/managed-repo.md b/website/docs/faqs/Git/managed-repo.md
new file mode 100644
index 00000000000..c68bd5ab896
--- /dev/null
+++ b/website/docs/faqs/Git/managed-repo.md
@@ -0,0 +1,10 @@
+---
+title: How can we move our project from a managed repository, to a self-hosted repository?
+description: "Email Support to request your managed repository"
+sidebar_label: 'How to request your managed repository zipfile'
+id: managed-repo
+---
+
+dbt Labs can send your managed repository through a ZIP file in its current state for you to push up to a git provider. After that, you'd just need to switch over to the [repo in your project](/docs/collaborate/git/import-a-project-by-git-url) to point to the new repository.
+
+When you're ready to do this, [contact the dbt Labs Support team](mailto:support@getdbt.com) with your request and your managed repo URL, which you can find by navigating to your project setting. To find project settings, click the gear icon in the upper right, select **Account settings**, click **Projects**, and then select your project. Under **Repository** in the project details page, you can find your managed repo URL.
diff --git a/website/docs/faqs/Git/run-on-pull.md b/website/docs/faqs/Git/run-on-pull.md
new file mode 100644
index 00000000000..3536259bb79
--- /dev/null
+++ b/website/docs/faqs/Git/run-on-pull.md
@@ -0,0 +1,15 @@
+---
+title: Why is Run on Pull request grayed out?
+description: "Use the GitHub auth method to enable Rull on Pull request"
+sidebar_label: 'Run on Pull request grayed out'
+id: run-on-pull
+---
+
+If you're unable to enable Run on Pull requests, you'll want to make sure your existing repo was not added via the Deploy Key auth method.
+
+If it was added via a deploy key method, you'll want to use the [GitHub auth method](https://docs.getdbt.com/docs/cloud-installing-the-github-application) to enable CI in dbt Cloud.
+
+To go ahead and enable 'Run on Pull requests', you'll want to remove dbt Cloud from the Apps & Integration on GitHub and re-integrate it again via the GitHub app method.
+
+If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
+
diff --git a/website/docs/faqs/jinja-whitespace.md b/website/docs/faqs/Jinja/jinja-whitespace.md
similarity index 82%
rename from website/docs/faqs/jinja-whitespace.md
rename to website/docs/faqs/Jinja/jinja-whitespace.md
index cd26398a2b9..90a7dc5d2e1 100644
--- a/website/docs/faqs/jinja-whitespace.md
+++ b/website/docs/faqs/Jinja/jinja-whitespace.md
@@ -1,6 +1,10 @@
---
title: My compiled SQL has a lot of spaces and new lines, how can I get rid of it?
+description: "Managing whitespace control"
+sidebar_label: 'Compiled sql has a lot of white space'
+id: jinja-whitespace
---
+
This is known as "whitespace control".
Use a minus sign (`-`, e.g. `{{- ... -}}`, `{%- ... %}`, `{#- ... -#}`) at the start or end of a block to strip whitespace before or after the block (more docs [here](https://jinja.palletsprojects.com/page/templates/#whitespace-control)). Check out the [tutorial on using Jinja](using-jinja#use-whitespace-control-to-tidy-up-compiled-code) for an example.
diff --git a/website/docs/faqs/quoting-column-names.md b/website/docs/faqs/Jinja/quoting-column-names.md
similarity index 87%
rename from website/docs/faqs/quoting-column-names.md
rename to website/docs/faqs/Jinja/quoting-column-names.md
index ccca7566dc4..aff4a36b886 100644
--- a/website/docs/faqs/quoting-column-names.md
+++ b/website/docs/faqs/Jinja/quoting-column-names.md
@@ -1,10 +1,16 @@
---
title: Why do I need to quote column names in Jinja?
+description: "Use quotes to pass string"
+sidebar_label: 'Why quote column names in Jinja'
+id: quoting-column-names
---
+
In the [macro example](jinja-macros#macros) we passed the column name `amount` quotes:
+
```sql
{{ cents_to_dollars('amount') }} as amount_usd
```
+
We have to use quotes to pass the _string_ `'amount'` to the macro.
Without the quotes, the Jinja parser will look for a variable named `amount`. Since this doesn't exist, it will compile to nothing.
diff --git a/website/docs/faqs/which-jinja-docs.md b/website/docs/faqs/Jinja/which-jinja-docs.md
similarity index 73%
rename from website/docs/faqs/which-jinja-docs.md
rename to website/docs/faqs/Jinja/which-jinja-docs.md
index 01c17f50954..bbb68318ffb 100644
--- a/website/docs/faqs/which-jinja-docs.md
+++ b/website/docs/faqs/Jinja/which-jinja-docs.md
@@ -1,9 +1,12 @@
---
title: Which docs should I use when writing Jinja or creating a macro?
+description: "Useful Jinja docs"
+sidebar_label: 'Useful Jinja docs'
+id: which-jinja-docs
---
If you are stuck with a Jinja issue, it can get confusing where to check for more information. We recommend you check (in order):
1. [Jinja's Template Designer Docs](https://jinja.palletsprojects.com/page/templates/): This is the best reference for most of the Jinja you'll use
-2. [Our Jinja function reference](dbt-jinja-functions): This documents any additional functionality we've added to Jinja in dbt.
+2. [Our Jinja function reference](/reference/dbt-jinja-functions): This documents any additional functionality we've added to Jinja in dbt.
3. [Agate's table docs](https://agate.readthedocs.io/page/api/table.html): If you're operating on the result of a query, dbt will pass it back to you as an agate table. This means that the methods you call on the materializations : `view`, `table`, `incremental` and `ephemeral`. Check out the documentation on [materializations](materializations) for more information on each of these options.
diff --git a/website/docs/faqs/configurable-model-path.md b/website/docs/faqs/Models/configurable-model-path.md
similarity index 82%
rename from website/docs/faqs/configurable-model-path.md
rename to website/docs/faqs/Models/configurable-model-path.md
index 39933c88dd2..9d16bb1f05f 100644
--- a/website/docs/faqs/configurable-model-path.md
+++ b/website/docs/faqs/Models/configurable-model-path.md
@@ -1,5 +1,9 @@
---
title: Can I store my models in a directory other than the `models` directory in my project?
+description: "How to name models directory"
+sidebar_label: 'How to name models directory'
+id: configurable-model-path
+
---
diff --git a/website/docs/faqs/Models/create-a-schema.md b/website/docs/faqs/Models/create-a-schema.md
new file mode 100644
index 00000000000..4bac0cfcd53
--- /dev/null
+++ b/website/docs/faqs/Models/create-a-schema.md
@@ -0,0 +1,8 @@
+---
+title: Do I need to create my target schema before running dbt?
+description: "Target schema is checked or created for you before running dbt"
+sidebar_label: 'Target schema before running dbt'
+id: create-a-schema
+
+---
+Nope! dbt will check if the schema exists when it runs. If the schema does not exist, dbt will create it for you.
diff --git a/website/docs/faqs/create-dependencies.md b/website/docs/faqs/Models/create-dependencies.md
similarity index 83%
rename from website/docs/faqs/create-dependencies.md
rename to website/docs/faqs/Models/create-dependencies.md
index a436ec8066c..4a84e3ab7d7 100644
--- a/website/docs/faqs/create-dependencies.md
+++ b/website/docs/faqs/Models/create-dependencies.md
@@ -1,5 +1,9 @@
---
title: How do I create dependencies between models?
+description: "Using ref function to create dependencies"
+sidebar_label: 'Creating dependencies between models'
+id: create-dependencies
+
---
When you use the `ref` [function](dbt-jinja-functions/ref), dbt automatically infers the dependencies between models.
@@ -40,4 +44,4 @@ Found 2 models, 28 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 0 s
Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
```
-To get some practice with this, we recommend you complete the [tutorial](tutorial/getting-started.md) to build your first dbt project
+To learn more about building a dbt project, we recommend you complete the [Getting Started guide](/docs/get-started/getting-started/overview).
diff --git a/website/docs/faqs/insert-records.md b/website/docs/faqs/Models/insert-records.md
similarity index 53%
rename from website/docs/faqs/insert-records.md
rename to website/docs/faqs/Models/insert-records.md
index d6aee7226c9..d522f6fbc85 100644
--- a/website/docs/faqs/insert-records.md
+++ b/website/docs/faqs/Models/insert-records.md
@@ -1,8 +1,12 @@
---
title: If models can only be `select` statements, how do I insert records?
+description: "Incremental models to insert records"
+sidebar_label: 'How to insert records'
+id: insert-records
+
---
For those coming from an
-
```sql
diff --git a/website/docs/faqs/Models/reference-models-in-another-project.md b/website/docs/faqs/Models/reference-models-in-another-project.md
new file mode 100644
index 00000000000..0a06b04e7bd
--- /dev/null
+++ b/website/docs/faqs/Models/reference-models-in-another-project.md
@@ -0,0 +1,11 @@
+---
+title: How can I reference models or macros in another project?
+description: "Use packages to add another project to your dbt project"
+sidebar_label: 'Reference models or macros in another project'
+id: reference-models-in-another-project
+
+---
+
+You can use [packages]/docs/build/packages) to add another project to your dbt
+project, including other projects you've created. Check out the [docs](/docs/build/packages)
+for more information!
diff --git a/website/docs/faqs/removing-deleted-models.md b/website/docs/faqs/Models/removing-deleted-models.md
similarity index 76%
rename from website/docs/faqs/removing-deleted-models.md
rename to website/docs/faqs/Models/removing-deleted-models.md
index 150d10237a1..05682465dc0 100644
--- a/website/docs/faqs/removing-deleted-models.md
+++ b/website/docs/faqs/Models/removing-deleted-models.md
@@ -1,5 +1,9 @@
---
title: How do I remove deleted models from my data warehouse?
+description: "Remove deleted models directly in your data warehouse"
+sidebar_label: 'Removing deleted models from your warehouse'
+id: removing-deleted-models
+
---
If you delete a model from your dbt project, dbt does not automatically drop the relation from your schema. This means that you can end up with extra objects in schemas that dbt creates, which can be confusing to other users.
diff --git a/website/docs/faqs/run-downtime.md b/website/docs/faqs/Models/run-downtime.md
similarity index 59%
rename from website/docs/faqs/run-downtime.md
rename to website/docs/faqs/Models/run-downtime.md
index 5b976bde258..4c19511061b 100644
--- a/website/docs/faqs/run-downtime.md
+++ b/website/docs/faqs/Models/run-downtime.md
@@ -1,6 +1,10 @@
---
title: If I rerun dbt, will there be any downtime as models are rebuilt?
+description: "No downtime when re-building models in dbt "
+sidebar_label: 'Re-building models in dbt atomically'
+id: run-downtime
+
---
Nope! The SQL that dbt generates behind the scenes ensures that any relations are replaced atomically (i.e. your business users won't experience any downtime).
-The implementation of this varies on each warehouse, check out the [logs](faqs/checking-logs.md) to see the SQL dbt is executing.
+The implementation of this varies on each warehouse, check out the [logs](/docs/faqs/Runs/checking-logs) to see the SQL dbt is executing.
diff --git a/website/docs/faqs/source-quotes.md b/website/docs/faqs/Models/source-quotes.md
similarity index 86%
rename from website/docs/faqs/source-quotes.md
rename to website/docs/faqs/Models/source-quotes.md
index 5b010dc1e70..da5a278669b 100644
--- a/website/docs/faqs/source-quotes.md
+++ b/website/docs/faqs/Models/source-quotes.md
@@ -1,5 +1,9 @@
---
title: I need to use quotes to select from my source, what should I do?
+description: "Use quoting property to quote values"
+sidebar_label: 'How to quote values'
+id: source-quotes
+
---
This is reasonably common on Snowflake in particular.
diff --git a/website/docs/faqs/specifying-column-types.md b/website/docs/faqs/Models/specifying-column-types.md
similarity index 82%
rename from website/docs/faqs/specifying-column-types.md
rename to website/docs/faqs/Models/specifying-column-types.md
index 034df62abcb..1adaf30fdce 100644
--- a/website/docs/faqs/specifying-column-types.md
+++ b/website/docs/faqs/Models/specifying-column-types.md
@@ -1,5 +1,9 @@
---
title: How do I specify column types?
+description: "Specify column types in models"
+sidebar_label: 'Specify column types in models'
+id: specifying-column-types
+
---
Simply cast the column to the correct type in your model:
@@ -33,6 +37,7 @@ create table dbt_alice.my_table as (
So long as your model queries return the correct column type, the table you create will also have the correct column type.
To define additional column options:
-* Rather than enforcing uniqueness and not-null constraints on your column, use dbt's [testing](building-a-dbt-project/tests) functionality to check that your assertions about your model hold true.
+
+* Rather than enforcing uniqueness and not-null constraints on your column, use dbt's [testing](/docs/build/tests) functionality to check that your assertions about your model hold true.
* Rather than creating default values for a column, use SQL to express defaults (e.g. `coalesce(updated_at, current_timestamp()) as updated_at`)
* In edge-cases where you _do_ need to alter a column (e.g. column-level encoding on Redshift), consider implementing this via a [post-hook](pre-hook-post-hook).
diff --git a/website/docs/faqs/sql-dialect.md b/website/docs/faqs/Models/sql-dialect.md
similarity index 86%
rename from website/docs/faqs/sql-dialect.md
rename to website/docs/faqs/Models/sql-dialect.md
index a8885a70fda..df7168a1b60 100644
--- a/website/docs/faqs/sql-dialect.md
+++ b/website/docs/faqs/Models/sql-dialect.md
@@ -1,14 +1,15 @@
---
-title: Which SQL dialect should I write my models in?
+title: Which SQL dialect should I write my models in? Or which SQL dialect does dbt use?
+description: "Use SQL dialect of your own database"
+sidebar_label: 'Which SQL dialect to use?'
+id: sql-dialect
---
-Or:
-### Which SQL dialect does dbt use?
dbt can feel like magic, but it isn't actually magic. Under the hood, it's running SQL in your own warehouse — your data is not processed outside of your warehouse.
As such, your models should just use the **SQL dialect of your own database**. Then, when dbt wraps your `select` statements in the appropriate /` and the logs in `logs/dbt.log` to see what dbt is running behind the scenes.
diff --git a/website/docs/faqs/define-a-column-type.md b/website/docs/faqs/Project/define-a-column-type.md
similarity index 87%
rename from website/docs/faqs/define-a-column-type.md
rename to website/docs/faqs/Project/define-a-column-type.md
index 13f2844bc6c..0f15f9e6892 100644
--- a/website/docs/faqs/define-a-column-type.md
+++ b/website/docs/faqs/Project/define-a-column-type.md
@@ -1,5 +1,9 @@
---
title: How do I define a column type?
+description: "Using cast function to define a column type"
+sidebar_label: 'How to define a column type'
+id: define-a-column-type
+
---
Your warehouse's SQL engine automatically assigns a [datatype](https://www.w3schools.com/sql/sql_datatypes.asp) to every column, whether it's found in a source or model. To force SQL to treat a columns a certain datatype, use `cast` functions:
@@ -16,7 +20,7 @@ from {{ ref('stg_orders') }}
-Many modern data warehouses now support `::` syntax as a shorthand for `cast( as )`.
+Many modern
diff --git a/website/docs/faqs/Project/docs-for-multiple-projects.md b/website/docs/faqs/Project/docs-for-multiple-projects.md
new file mode 100644
index 00000000000..b7aa1452b39
--- /dev/null
+++ b/website/docs/faqs/Project/docs-for-multiple-projects.md
@@ -0,0 +1,11 @@
+---
+title: Can I render docs for multiple projects?
+description: "Using packages to render docs for multiple projects"
+sidebar_label: 'Render docs for multiple projects'
+id: docs-for-multiple-projects
+
+---
+
+Yes! To do this, you'll need to create a "super project" that lists each project as a dependent [package](/docs/build/packages) in a `packages.yml` file. Then run `dbt deps` to install the projects as packages, prior to running `dbt docs generate`.
+
+If you are going down the route of multiple projects, be sure to check out our advice [1](https://discourse.getdbt.com/t/should-i-have-an-organisation-wide-project-a-monorepo-or-should-each-work-flow-have-their-own/666) [2](https://discourse.getdbt.com/t/how-to-configure-your-dbt-repository-one-or-many/2121) on the topic.
diff --git a/website/docs/faqs/example-projects.md b/website/docs/faqs/Project/example-projects.md
similarity index 76%
rename from website/docs/faqs/example-projects.md
rename to website/docs/faqs/Project/example-projects.md
index 45aed62309b..d4bebc7206d 100644
--- a/website/docs/faqs/example-projects.md
+++ b/website/docs/faqs/Project/example-projects.md
@@ -1,13 +1,17 @@
---
title: Are there any example dbt projects?
+description: "List of example dbt projects"
+sidebar_label: 'dbt project examples'
+id: example-projects
+
---
Yes!
-* **Getting Started Tutorial:** You can build your own example dbt project in the [Getting Started Tutorial](tutorial/getting-started.md)
+* **Getting Started Tutorial:** You can build your own example dbt project in the [Getting Started guide](/docs/get-started/getting-started/overview)
* **Jaffle Shop:** A demonstration project (closely related to the tutorial) for a fictional ecommerce store ([source code](https://github.com/dbt-labs/jaffle_shop))
* **MRR Playbook:** A demonstration project that models subscription revenue ([source code](https://github.com/dbt-labs/mrr-playbook), [docs](https://www.getdbt.com/mrr-playbook/#!/overview))
* **Attribution Playbook:** A demonstration project that models marketing attribution ([source code](https://github.com/dbt-labs/attribution-playbook), [docs](https://www.getdbt.com/attribution-playbook/#!/overview))
* **GitLab:** Gitlab's internal dbt project is open source and is a great example of how to use dbt at scale ([source code](https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt), [docs](https://dbt.gitlabdata.com/))
-If you have an example project to add to this list, suggest an [edit](https://github.com/dbt-labs/docs.getdbt.com/edit/master/website/docs/faqs/example-projects.md).
+If you have an example project to add to this list, suggest an edit by clicking **Edit this page** below.
diff --git a/website/docs/faqs/exclude-table-from-freshness.md b/website/docs/faqs/Project/exclude-table-from-freshness.md
similarity index 82%
rename from website/docs/faqs/exclude-table-from-freshness.md
rename to website/docs/faqs/Project/exclude-table-from-freshness.md
index 17a21940e44..c8aacce2de9 100644
--- a/website/docs/faqs/exclude-table-from-freshness.md
+++ b/website/docs/faqs/Project/exclude-table-from-freshness.md
@@ -1,5 +1,9 @@
---
title: How do I exclude a table from a freshness snapshot?
+description: "Use null to exclude a table from a freshness snapshot"
+sidebar_label: 'Exclude a table from freshness snapshot '
+id: exclude-table-from-freshness
+
---
Some tables in a data source may be updated infrequently. If you've set a `freshness` property at the source level, this views , and then change models to tables when required for performance reasons (i.e. downstream queries have slowed).
diff --git a/website/docs/faqs/which-schema.md b/website/docs/faqs/Project/which-schema.md
similarity index 69%
rename from website/docs/faqs/which-schema.md
rename to website/docs/faqs/Project/which-schema.md
index 0378410b19f..f0634ac8c85 100644
--- a/website/docs/faqs/which-schema.md
+++ b/website/docs/faqs/Project/which-schema.md
@@ -1,10 +1,14 @@
---
title: How did dbt choose which schema to build my models in?
+description: "You can change your target schema in your Environment Settings."
+sidebar_label: 'dbt builds models in your target schema'
+id: which-schema
+
---
By default, dbt builds models in your target schema. To change your target schema:
* If you're developing in **dbt Cloud**, these are set for each user when you first use a development environment.
* If you're developing with the **dbt CLI**, this is the `schema:` parameter in your `profiles.yml` file.
-If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](using-custom-schemas).
+If you wish to split your models across multiple schemas, check out the docs on [using custom schemas](/docs/build/custom-schemas).
-Note: on BigQuery, `dataset` is used interchangeably with `schema`.
\ No newline at end of file
+Note: on BigQuery, `dataset` is used interchangeably with `schema`.
diff --git a/website/docs/faqs/why-not-write-dml.md b/website/docs/faqs/Project/why-not-write-dml.md
similarity index 86%
rename from website/docs/faqs/why-not-write-dml.md
rename to website/docs/faqs/Project/why-not-write-dml.md
index a553460de57..cf778c2086c 100644
--- a/website/docs/faqs/why-not-write-dml.md
+++ b/website/docs/faqs/Project/why-not-write-dml.md
@@ -1,14 +1,19 @@
---
-title: Why can't I just write DML in my transformations?
+title: "Why can't I just write DML in my transformations?"
+description: "Using SQL in your transformations instead of DML."
+sidebar_label: 'Why not write in DML'
+id: why-not-write-dml
+
---
-Or:
-### I'm already familiar with DML, and can write these statements manually, why should I use dbt to do this?
#### `select` statements make transformations accessible
+
More people know how to write `select` statements, than
-```
+```shell
$ dbt seed
Running with dbt=0.16.0-rc2
Found 0 models, 0 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 1 seed file, 0 sources
@@ -37,7 +41,7 @@ Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
-```
+```shell
$ dbt seed
Running with dbt=0.16.0-rc2
Found 0 models, 0 tests, 0 snapshots, 0 analyses, 149 macros, 0 operations, 1 seed file, 0 sources
@@ -62,7 +66,8 @@ Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
In this case, you should rerun the command with a `--full-refresh` flag, like so:
-```
+
+```shell
dbt seed --full-refresh
```
diff --git a/website/docs/faqs/leading-zeros-in-seed.md b/website/docs/faqs/Seeds/leading-zeros-in-seed.md
similarity index 75%
rename from website/docs/faqs/leading-zeros-in-seed.md
rename to website/docs/faqs/Seeds/leading-zeros-in-seed.md
index c59e663c9cc..f2ffeb05f3f 100644
--- a/website/docs/faqs/leading-zeros-in-seed.md
+++ b/website/docs/faqs/Seeds/leading-zeros-in-seed.md
@@ -1,5 +1,9 @@
---
title: How do I preserve leading zeros in a seed?
+description: "Use column types to include leading zeros in seed"
+sidebar_label: 'Include leading zeroes in your seed file'
+id: leading-zeros-in-seed
+
---
If you need to preserve leading zeros (for example in a zipcode or mobile number):
diff --git a/website/docs/faqs/load-raw-data-with-seed.md b/website/docs/faqs/Seeds/load-raw-data-with-seed.md
similarity index 72%
rename from website/docs/faqs/load-raw-data-with-seed.md
rename to website/docs/faqs/Seeds/load-raw-data-with-seed.md
index 142ce37979e..e33780f5dcc 100644
--- a/website/docs/faqs/load-raw-data-with-seed.md
+++ b/website/docs/faqs/Seeds/load-raw-data-with-seed.md
@@ -1,9 +1,13 @@
---
title: Can I use seeds to load raw data?
+description: "Use seeds to load business specific logic"
+sidebar_label: 'Seed data files requirements'
+id: load-raw-data-with-seed
+
---
Seeds should **not** be used to load raw data (for example, large CSV exports from a production database).
Since seeds are version controlled, they are best suited to files that contain business-specific logic, for example a list of country codes or user IDs of employees.
-Loading CSVs using dbt's seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your data warehouse.
+Loading CSVs using dbt's seed functionality is not performant for large files. Consider using a different tool to load these CSVs into your
-- **v1.0.0:** The config 'data-paths' has been deprecated in favor of [`seed-paths`](seed-paths).
+- **v1.0.0:** The config 'data-paths' has been deprecated in favor of [`seed-paths`](seed-paths).
diff --git a/website/docs/faqs/configurable-data-test-path.md b/website/docs/faqs/Tests/configurable-data-test-path.md
similarity index 82%
rename from website/docs/faqs/configurable-data-test-path.md
rename to website/docs/faqs/Tests/configurable-data-test-path.md
index e4bda68b810..2c3e8d729ba 100644
--- a/website/docs/faqs/configurable-data-test-path.md
+++ b/website/docs/faqs/Tests/configurable-data-test-path.md
@@ -1,5 +1,9 @@
---
title: Can I store my tests in a directory other than the `tests` directory in my project?
+description: "Where to store tests in a directory"
+sidebar_label: 'How to name tests directory'
+id: configurable-data-test-path
+
---
By default, dbt expects your singular test files to be located in the `tests` subdirectory of your project, and generic test definitions to be located in `tests/generic` or `macros`.
diff --git a/website/docs/faqs/custom-test-thresholds.md b/website/docs/faqs/Tests/custom-test-thresholds.md
similarity index 79%
rename from website/docs/faqs/custom-test-thresholds.md
rename to website/docs/faqs/Tests/custom-test-thresholds.md
index b84dc14d4cd..14dd36666c0 100644
--- a/website/docs/faqs/custom-test-thresholds.md
+++ b/website/docs/faqs/Tests/custom-test-thresholds.md
@@ -1,9 +1,14 @@
---
title: Can I set test failure thresholds?
+description: "Use configs to set custom failure thresholds in tests"
+sidebar_label: 'How to set failure thresholds in test'
+id: custom-test-thresholds
+
---
As of `v0.20.0`, you can use the `error_if` and `warn_if` configs to set custom failure thresholds in your tests. For more details, see [reference](reference/resource-configs/severity) for more information.
For dbt `v0.19.0` and earlier, you could try these possible solutions:
+
* Setting the [severity](resource-properties/tests#severity) to `warn`, or:
* Writing a [custom generic test](custom-generic-tests) that accepts a threshold argument ([example](https://discourse.getdbt.com/t/creating-an-error-threshold-for-schema-tests/966))
diff --git a/website/docs/faqs/multiple-test-files.md b/website/docs/faqs/Tests/multiple-test-files.m
similarity index 60%
rename from website/docs/faqs/multiple-test-files.md
rename to website/docs/faqs/Tests/multiple-test-files.m
index 04e62971400..2ae6f4f93d3 100644
--- a/website/docs/faqs/multiple-test-files.md
+++ b/website/docs/faqs/Tests/multiple-test-files.m
@@ -1,4 +1,8 @@
---
title: Do all my tests go in one file?
+description: "Structuring files for tests"
+sidebar_label: 'How to structure files for tests'
+id: multiple-test-files
+
---
No! You can use as many files as you want! Some folks find it useful to have one file per model, we tend to have one per directory.
diff --git a/website/docs/faqs/recommended-tests.md b/website/docs/faqs/Tests/recommended-tests.md
similarity index 61%
rename from website/docs/faqs/recommended-tests.md
rename to website/docs/faqs/Tests/recommended-tests.md
index 8ed2e736307..d79fb173e9e 100644
--- a/website/docs/faqs/recommended-tests.md
+++ b/website/docs/faqs/Tests/recommended-tests.md
@@ -1,8 +1,12 @@
---
title: What tests should I add to my project?
+description: "Recommended tests for project"
+sidebar_label: 'Recommended tests for project'
+id: recommended-tests
+
---
We recommend that every model has a test on a The `seeds:` key is new in 0.16.0. Prior to this, use a `models:` key instead.
To test and document seeds, use a [schema file](configs-and-properties) and nest the configurations under a `seeds:` key
-## Example:
+## Example
diff --git a/website/docs/faqs/testing-sources.md b/website/docs/faqs/Tests/testing-sources.md
similarity index 71%
rename from website/docs/faqs/testing-sources.md
rename to website/docs/faqs/Tests/testing-sources.md
index 2f2603856e4..06051eae7e9 100644
--- a/website/docs/faqs/testing-sources.md
+++ b/website/docs/faqs/Tests/testing-sources.md
@@ -1,23 +1,28 @@
---
title: How do I run tests on just my sources?
+description: "Use the select source command to test sources"
+sidebar_label: 'Run tests on all sources'
+id: testing-sources
+
---
To run tests on all sources, use the following command:
-```
+```shell
$ dbt test --select source:*
```
+
(You can also use the `-s` shorthand here instead of `--select`)
To run tests on one source (and all of its tables):
-```
+```shell
$ dbt test --select source:jaffle_shop
```
And, to run tests on one source
diff --git a/website/docs/faqs/when-to-test.md b/website/docs/faqs/Tests/when-to-test.md
similarity index 72%
rename from website/docs/faqs/when-to-test.md
rename to website/docs/faqs/Tests/when-to-test.md
index 63585a333e8..fc97f98000b 100644
--- a/website/docs/faqs/when-to-test.md
+++ b/website/docs/faqs/Tests/when-to-test.md
@@ -1,5 +1,9 @@
---
title: When should I run my tests?
+description: "Run tests in production or when writing new code"
+sidebar_label: 'When to run tests'
+id: when-to-test
+
---
You should run your tests whenever you are writing new code (to ensure you haven't broken any existing models by changing SQL), and whenever you run your transformations in production (to ensure that your assumptions about your source data are still valid).
diff --git a/website/docs/faqs/Troubleshooting/access-gdrive-credential.md b/website/docs/faqs/Troubleshooting/access-gdrive-credential.md
new file mode 100644
index 00000000000..ca73c5c2631
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/access-gdrive-credential.md
@@ -0,0 +1,24 @@
+---
+title: I'm receiving a "Permission denied while getting Drive credential" error when trying to query from Google Drive?
+description: "Grant BigQuery service account access"
+sidebar_label: 'Error when trying to query from Google Drive'
+id: access-gdrive-credential
+
+---
+
+If you're seeing the below error when you try to query a dataset from a Google Drive document in the IDE, the IDE due to the below error message, we'll do our best to get you unstuck with the below steps!
+
+```
+Access denied: BigQuery BigQuery: Permission denied while getting Drive credentials
+```
+
+Usually this errors indicates that you haven't granted the BigQuery service account access to the specific Google Drive document. If you're seeing this error, try giving the service account (client email seen [here](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-your-database#connecting-to-bigquery)) you are using for your BigQuery connection in dbt Cloud, permission to your Google Drive or Google Sheet. You'll want to do this directly in your Google Document and click the 'share' button and enter the client email there.
+
+If you are experiencing this error when using oAuth, and you have verified your access to the Google Sheet, you may need to grant permissions for gcloud to access Google Drive:
+
+```
+gcloud auth application-default login --scopes=openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/sqlservice.login,https://www.googleapis.com/auth/drive
+```
+For more info see the [gcloud auth application-default documentation](https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login)
+
+If you've tried the steps above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Troubleshooting/access_token_error.md b/website/docs/faqs/Troubleshooting/access_token_error.md
new file mode 100644
index 00000000000..85be1fcd9dc
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/access_token_error.md
@@ -0,0 +1,21 @@
+---
+title: I'm receiving an `access_token` error when trying to run queries in the IDE.
+description: "Reauthenticate warehouse when seeing `access_token` error"
+sidebar_label: '`access_token` error in the IDE'
+id: access_token_error
+
+---
+
+If you're seeing the below error when you try to run queries in the IDE, have no fear - we'll do our best to get you unstuck with the following steps!
+
+ `access_token`
+
+In order to resolve the issue, you'll want to:
+
+1. Go to you Profile (upper right corner of dbt Cloud)
+2. Click on the correct warehouse connection under "Credentials"
+3. Click "Connect Snowflake Account" green button in the "Development Credentials" section. This drives you through reauthentication through the SSO flow.
+
+If you've tried the step above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
+
+
diff --git a/website/docs/faqs/dispatch-could-not-find-package.md b/website/docs/faqs/Troubleshooting/dispatch-could-not-find-package.md
similarity index 60%
rename from website/docs/faqs/dispatch-could-not-find-package.md
rename to website/docs/faqs/Troubleshooting/dispatch-could-not-find-package.md
index dea47e27f98..8b9f6c15367 100644
--- a/website/docs/faqs/dispatch-could-not-find-package.md
+++ b/website/docs/faqs/Troubleshooting/dispatch-could-not-find-package.md
@@ -1,10 +1,18 @@
---
-title: "[Error] Could not find package 'my_project'"
+title: "[Error] Could not find my_project package"
+description: "Macros missing from packages"
+sidebar_label: 'Could not find package error'
+id: dispatch-could-not-find-package
+
---
If a package name is included in the `search_order` of a project-level `dispatch` config, dbt expects that package to contain macros which are viable candidates for dispatching. If an included package does not contain _any_ macros, dbt will raise an error like:
-```
+
+```shell
Compilation Error
In dispatch: Could not find package 'my_project'
```
+
This does not mean the package or root project is missing—it means that any macros from it are missing, and so it is missing from the search spaces available to `dispatch`.
+
+If you've tried the step above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Troubleshooting/git-revlist-error.md b/website/docs/faqs/Troubleshooting/git-revlist-error.md
new file mode 100644
index 00000000000..e90c2729ec5
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/git-revlist-error.md
@@ -0,0 +1,31 @@
+---
+title: I'm receiving a git rev-list master error in the IDE?
+description: "Primary branch not recognized"
+sidebar_label: 'git rev-list master error in the IDE'
+id: git-revlist-error
+---
+
+If you're unable to access the IDE due to the below error message, we'll do our best to get you unstuck with the below steps!
+
+```shell
+git rev-list master..origin/main --count
+fatal: ambiguous argument 'master..origin/main': unknown revision or path not in the working tree.
+Use '--' to separate paths from revisions, like this:
+'git [...] -- [...]'
+```
+
+Usually this error indicates that the "main" branch name has changed or it is possible that dbt Cloud was unable to determine what your primary branch was. No worries, we have a few workarounds for you to try:
+
+**Workaround 1**
+Take a look at your Environment Settings - If you **do not** have a custom branch filled in your Environment Settings:
+
+1. Disconnect and reconnect your repository [connection](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/cloud-import-a-project-by-git-url) on your Project Settings page. This should then allow dbt Cloud to pick up that the "main" branch is now called `main`
+2. In the Environment Settings, set the custom branch to 'master' and refresh the IDE
+
+**Workaround 2**
+Take a look at your Environment Settings - If you **do** have a custom branch filled in your Environment Settings:
+
+1. Disconnecting and reconnecting your repository [connection](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/cloud-import-a-project-by-git-url) on your Project Settings page. This should then allow dbt Cloud to pick up that the "main" branch is now called `main`
+2. In the Environment Settings, remove the custom branch and refresh the IDE
+
+If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Troubleshooting/gitignore.md b/website/docs/faqs/Troubleshooting/gitignore.md
new file mode 100644
index 00000000000..47c7500e662
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/gitignore.md
@@ -0,0 +1,26 @@
+---
+title: Why can't I checkout a branch or create a new branch?
+description: "Add or fill in gitignore file"
+sidebar_label: 'Unable to checkout or create branch'
+id: gitignore
+---
+
+If you're finding yourself unable to revert changes, check out a branch or click commit - this is usually do to your project missing a .[gitignore](https://github.com/dbt-labs/dbt-starter-project/blob/main/.gitignore) file OR your gitignore file doesn't contain the necessary content inside the folder.
+
+This is what causes that 'commit' git action button to display. No worries though - to fix this, you'll need to complete the following steps in order:
+
+1. In the Cloud IDE, add the missing .gitignore file or contents to your project. You'll want to make sure the .gitignore file includes the following:
+
+ ```shell
+ target/
+ dbt_modules/
+ dbt_packages/
+ logs/
+ ```
+
+2. Once you've added that, make sure to save and commit.
+
+3. Navigate to the same branch in your remote repository (which can be accessed directly through your git provider's web interface) and delete the logs, target, and dbt_modules/dbt_packages folders.
+
+4. Go back into the Cloud IDE and reclone your repository. This can be done by clicking on the green "ready" in the bottom right corner of the IDE (next to the command bar), and then clicking the orange "reclone repo" button in the pop up.
+
diff --git a/website/docs/faqs/Troubleshooting/gitlab-authentication.md b/website/docs/faqs/Troubleshooting/gitlab-authentication.md
new file mode 100644
index 00000000000..7e7177b022a
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/gitlab-authentication.md
@@ -0,0 +1,29 @@
+---
+title: I'm seeing a Gitlab authentication out of date error loop
+description: "GitLab and dbt Cloud deploy key mismatch "
+sidebar_label: 'GitLab authentication out of date'
+id: gitlab-authentication
+---
+
+If you're seeing a 'GitLab Authentication is out of date' 500 server error page - this usually occurs when the deploy key in the repository settings in both dbt Cloud and GitLab do not match.
+
+No worries - this is a current issue the dbt Labs team is working on and we have a few workarounds for you to try:
+
+### 1st Workaround
+
+1. Disconnect repo from project in dbt Cloud.
+2. Go to Gitlab and click on Settings > Repository.
+3. Under Repository Settings, remove/revoke active dbt Cloud deploy tokens and deploy keys.
+4. Attempt to reconnect your repository via dbt Cloud.
+5. You would then need to check Gitlab to make sure that the new deploy key is added.
+6. Once confirmed that it's added, refresh dbt Cloud and try developing once again.
+
+### 2nd Workaround
+
+1. Keep repo in project as is -- don't disconnect.
+2. Copy the deploy key generated in dbt Cloud.
+3. Go to Gitlab and click on Settings > Repository.
+4. Under Repository Settings, manually add to your Gitlab project deploy key repo (with `Grant write permissions` box checked).
+5. Go back to dbt Cloud, refresh your page and try developing again.
+
+If you've tried the workarounds above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Troubleshooting/nonetype-ide-error.md b/website/docs/faqs/Troubleshooting/nonetype-ide-error.md
new file mode 100644
index 00000000000..85e0c67ec6a
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/nonetype-ide-error.md
@@ -0,0 +1,22 @@
+---
+title: I'm receiving a NoneType object has no attribute error in the IDE?
+description: "Copy SSH key to your warehouse"
+sidebar_label: 'NoneType error in the IDE'
+id: nonetype-ide-error
+
+---
+
+If you're unable to access the IDE due to the below error message, we'll do our best to get you unstuck with the below steps!
+
+```shell
+NoneType object has no attribute
+enumerate_fields'
+```
+
+Usually this errors indicates that you tried connecting your database via [SSH tunnel](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-your-database#connecting-via-an-ssh-tunnel). If you're seeing this error, double-check you have supplied the following items:
+
+- the hostname
+- username
+- port of bastion server
+
+If you've tried the step above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Troubleshooting/partial-parsing-error.md b/website/docs/faqs/Troubleshooting/partial-parsing-error.md
new file mode 100644
index 00000000000..3f37928771f
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/partial-parsing-error.md
@@ -0,0 +1,11 @@
+---
+title: "I'm getting a \"Partial parsing enabled: 1 files deleted, 0 files added, 2 files changed\" compilation error in dbt Cloud?"
+description: "Delete partial_parse file"
+sidebar_label: 'partial_parse error in the IDE'
+id: partial-parsing-error
+
+---
+
+If you're receiving this error, try deleting the `target/partial_parse.msgpack` file from your project and refresh your IDE.
+
+If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Troubleshooting/runtime-error-could-not-find-profile.md b/website/docs/faqs/Troubleshooting/runtime-error-could-not-find-profile.md
new file mode 100644
index 00000000000..7d7ad21e4f6
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/runtime-error-could-not-find-profile.md
@@ -0,0 +1,28 @@
+---
+title: I'm receiving a Runtime Error Could not find profile named 'user' error?
+description: "Re-authorize your credentials on Profile Settings"
+sidebar_label: '"Could not find profile named user" error in the IDE'
+id: runtime-error-could-not-find-profile
+
+---
+
+If you're unable to access the IDE due to the below error message, we'll do our best to get you unstuck with the below steps!
+
+```shell
+Running with dbt=0.21.0
+Encountered an error while reading the project:
+ ERROR: Runtime Error
+ Could not find profile named 'user'
+Runtime Error
+ Could not run dbt'
+```
+
+Usually this errors indicates that there is an issue with missing/stale credentials/authentication. No worries, we have a few workarounds for you to try:
+
+**In the IDE:**
+If this is happening in the IDE, you'll want to navigate to the Profile settings where your development credentials are configured. Once you're there, you'll need to either re-enter or re-authorize your credentials in order to get around this error message.
+
+**In a job:**
+If this is happening in a job, it might be that you made some sort of change to the deployment environment in which the job is configured and did not re-enter your deployment credentials upon saving those changes. To fix this, you'll need to go back into the deployment environment settings, re-enter your credentials (either the private key/private key passphrase or the username and password), and kick off a new job run.
+
+If you've tried the step above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/Troubleshooting/runtime-packages.yml.md b/website/docs/faqs/Troubleshooting/runtime-packages.yml.md
new file mode 100644
index 00000000000..14e87be41d9
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/runtime-packages.yml.md
@@ -0,0 +1,28 @@
+---
+title: Why am I receiving a Runtime Error in my packages?
+description: "Update dbt_utils package in packages.yml file"
+sidebar_label: 'Runtime error in packages.yml file'
+id: runtime-packages.yml
+
+---
+
+If you're receiving the runtime error below in your packages.yml folder, it may be due to an old version of your dbt_utils package that isn't compatible with your current dbt Cloud version.
+
+```shell
+Running with dbt=xxx
+Runtime Error
+ Failed to read package: Runtime Error
+ Invalid config version: 1, expected 2
+ Error encountered in dbt_utils/dbt_project.yml
+ ```
+
+Try updating the old version of the dbt_utils package in your packages.yml to the latest version found in the [dbt hub](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/):
+
+```shell
+packages:
+- package: dbt-labs/dbt_utils
+
+version: xxx
+```
+
+If you've tried the workaround above and are still experiencing this behavior - reach out to the Support team at support@getdbt.com and we'll be happy to help!
diff --git a/website/docs/faqs/sql-errors.md b/website/docs/faqs/Troubleshooting/sql-errors.md
similarity index 69%
rename from website/docs/faqs/sql-errors.md
rename to website/docs/faqs/Troubleshooting/sql-errors.md
index 3e6b84b5842..b5101d3251a 100644
--- a/website/docs/faqs/sql-errors.md
+++ b/website/docs/faqs/Troubleshooting/sql-errors.md
@@ -1,11 +1,15 @@
---
-title: What happens if the SQL in my query is bad?
+title: What happens if the SQL in my query is bad or I get a database error?
+description: "Use error message and logs to debug"
+sidebar_label: 'How to debug SQL or database error'
+id: sql-errors
+
---
-Or:
-### I got a Database Error, what does that mean?
+
If there's a mistake in your SQL, dbt will return the error that your database returns.
-```shell-session
+
+```shell
$ dbt run --select customers
Running with dbt=0.15.0
Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 seed files, 0 sources
@@ -20,10 +24,10 @@ Found 3 models, 9 tests, 0 snapshots, 0 analyses, 133 macros, 0 operations, 0 se
Completed with 1 error and 0 warnings:
Database Error in model customers (models/customers.sql)
- Syntax error: Expected ")" but got identifier `grand-highway-265418` at [13:15]
+ Syntax error: Expected ")" but got identifier `your-info-12345` at [13:15]
compiled SQL at target/run/jaffle_shop/customers.sql
Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
```
-Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](faqs/checking-logs.md) to debug any errors.
+Any models downstream of this model will also be skipped. Use the error message and the [compiled SQL](/faqs/Runs/checking-logs) to debug any errors.
diff --git a/website/docs/faqs/Troubleshooting/unused-model-configurations.md b/website/docs/faqs/Troubleshooting/unused-model-configurations.md
new file mode 100644
index 00000000000..ba5506c260f
--- /dev/null
+++ b/website/docs/faqs/Troubleshooting/unused-model-configurations.md
@@ -0,0 +1,10 @@
+---
+title: I got an "unused model configurations" error message, what does this mean?
+description: "Model configurations error message"
+sidebar_label: 'unused model configurations error'
+id: unused-model-configurations
+
+---
+You might have forgotten to nest your configurations under your project name, or you might be trying to apply configurations to a directory that doesn't exist.
+
+Check out this [article](https://discourse.getdbt.com/t/faq-i-got-an-unused-model-configurations-error-message-what-does-this-mean/112) to understand more.
diff --git a/website/docs/faqs/bq-impersonate-service-account-setup.md b/website/docs/faqs/Warehouse/bq-impersonate-service-account-setup.md
similarity index 84%
rename from website/docs/faqs/bq-impersonate-service-account-setup.md
rename to website/docs/faqs/Warehouse/bq-impersonate-service-account-setup.md
index d57e23b32b5..60db41095e5 100644
--- a/website/docs/faqs/bq-impersonate-service-account-setup.md
+++ b/website/docs/faqs/Warehouse/bq-impersonate-service-account-setup.md
@@ -1,5 +1,9 @@
---
title: How can I set up the right permissions in BigQuery?
+description: "Use service account to set up permissions in BigQuery"
+sidebar_label: 'Setting up permissions in BigQuery"'
+id: bq-impersonate-service-account-setup
+
---
To use this functionality, first create the service account you want to
@@ -16,4 +20,4 @@ the [IAM Service Account Credentials API](https://console.cloud.google.com/apis/
Enabling the API and granting the role are eventually consistent operations,
taking up to 7 minutes to fully complete, but usually fully propagating within 60
seconds. Give it a few minutes, then add the `impersonate_service_account`
-option to your BigQuery profile configuration.
\ No newline at end of file
+option to your BigQuery profile configuration.
diff --git a/website/docs/faqs/bq-impersonate-service-account-why.md b/website/docs/faqs/Warehouse/bq-impersonate-service-account-why.md
similarity index 85%
rename from website/docs/faqs/bq-impersonate-service-account-why.md
rename to website/docs/faqs/Warehouse/bq-impersonate-service-account-why.md
index ffde684fed4..e56485e18e7 100644
--- a/website/docs/faqs/bq-impersonate-service-account-why.md
+++ b/website/docs/faqs/Warehouse/bq-impersonate-service-account-why.md
@@ -1,5 +1,9 @@
---
title: Why would I want to impersonate a service account?
+description: "Build models using dedicated service account"
+sidebar_label: 'Why impersonate a service account'
+id: bq-impersonate-service-account-why
+
---
You may want your models to be built using a dedicated service account that has
@@ -15,4 +19,4 @@ This allows you to reap the advantages of using federated identity for developer
directly, and without needing to create separate service account and keys for
each user. It also allows you to completely eliminate the need for service
account keys in CI as long as your CI is running on GCP (Cloud Build, Jenkins,
-GitLab/Github Runners, etc).
\ No newline at end of file
+GitLab/Github Runners, etc).
diff --git a/website/docs/faqs/connecting-to-two-dbs-not-allowed.md b/website/docs/faqs/Warehouse/connecting-to-two-dbs-not-allowed.md
similarity index 89%
rename from website/docs/faqs/connecting-to-two-dbs-not-allowed.md
rename to website/docs/faqs/Warehouse/connecting-to-two-dbs-not-allowed.md
index fb9ea905fa8..f7c546b3648 100644
--- a/website/docs/faqs/connecting-to-two-dbs-not-allowed.md
+++ b/website/docs/faqs/Warehouse/connecting-to-two-dbs-not-allowed.md
@@ -1,8 +1,13 @@
---
title: Can I connect my dbt project to two databases?
+description: "Prepare your warehouse for dbt transformations"
+sidebar_label: 'Can you connect dbt project to two databases?'
+id: connecting-to-two-dbs-not-allowed
+
---
The meaning of the term 'database' varies with each major warehouse manager. Hence, the answer to "can a dbt project connect to more than one database?" depends on the warehouse used in your tech stack.
+
* dbt projects connecting to warehouses like Snowflake or Bigquery—these empower one set of credentials to draw from all datasets or 'projects' available to an account—are _sometimes_ said to connect to more than one database.
* dbt projects connecting to warehouses like Redshift and Postgres—these tie one set of credentials to one database—are said to connect to one database only.
@@ -11,4 +16,3 @@ Sidestep the 'one database problem' by relying on materializations are custom Jinja _blocks_ that exist only in dbt.
diff --git a/website/docs/faqs/docs-for-multiple-projects.md b/website/docs/faqs/docs-for-multiple-projects.md
deleted file mode 100644
index 0966ec8d480..00000000000
--- a/website/docs/faqs/docs-for-multiple-projects.md
+++ /dev/null
@@ -1,7 +0,0 @@
----
-title: Can I render docs for multiple projects?
----
-
-Yes! To do this, you'll need to create a "super project" that lists each project as a dependent [package](package-management) in a `packages.yml` file. Then run `dbt deps` to install the projects as packages, prior to running `dbt docs generate`.
-
-If you are going down the route of multiple projects, be sure to check out our advice [1](https://discourse.getdbt.com/t/should-i-have-an-organisation-wide-project-a-monorepo-or-should-each-work-flow-have-their-own/666) [2](https://discourse.getdbt.com/t/how-to-configure-your-dbt-repository-one-or-many/2121)on the topic.
diff --git a/website/docs/faqs/failed-tests.md b/website/docs/faqs/failed-tests.md
deleted file mode 100644
index 3b4a1751648..00000000000
--- a/website/docs/faqs/failed-tests.md
+++ /dev/null
@@ -1,12 +0,0 @@
----
-title: One of my tests failed, how can I debug it?
----
-To debug a failing test, find the SQL that dbt ran by:
-* dbt Cloud:
- * Within the test output, click on the failed test, and then select "Details"
-* dbt CLI:
- * Open the file path returned as part of the error message.
- * Navigate to the `target/compiled/schema_tests` directory for all compiled test queries
-
-Copy the SQL into a query editor (in dbt Cloud, you can paste it into a new `Statement`), and run the query to find the records that failed.
-
diff --git a/website/docs/faqs/index.md b/website/docs/faqs/index.md
deleted file mode 100644
index 0a5777f1e53..00000000000
--- a/website/docs/faqs/index.md
+++ /dev/null
@@ -1,7 +0,0 @@
----
-slug: all
-id: all
-title: List of FAQs
----
-
-`IF EXISTS, DROP TABLE`
-
+
+
+
+#### Create a dbt Cloud account
-
-4. For the name, write `Databricks` or another simple title.
-5. For Databricks settings, reference your SQL endpoint connection details from step 6 of the previous section for each of the following fields:
+#### Connect dbt Cloud to Databricks
+
+1. Choose **Databricks** to setup your connection.
+
+
+
+4. For your Development Credentials, type:
+
+ - `User` and `token` that you saved in a previous step.
+ - You’ll notice that the schema name has been auto created for you. By convention, this is `dbt_
+
-
+
+
4. Set your development credentials. These credentials will be used by dbt Cloud to connect to Redshift. Those credentials (as provided in your cloudformation output) will be:
- Username: `dbtadmin`
- password: *this was the password that you set earlier in the guide*
- - schema: This is your sandbox schema where you will build all of your development objects into. We generally use the `dbt_
+
-4. Click **Test** at the top. This verifies that dbt Cloud can access your Redshift cluster.
-5. If you see "Connection test Succeeded!" then click **Continue**. If it fails, you might need to go back and check your Redshift settings and development credentials.
+4. Click **Test Connection** at the bottom. This verifies that dbt Cloud can access your Redshift cluster.
+5. If the connection test succeeds, click **Next**. If it fails, you may need to check your Redshift settings and credentials.
## Initialize your repository and start development
diff --git a/website/docs/tutorial/getting-set-up/setting-up-snowflake.md b/website/docs/docs/get-started/getting-started/getting-set-up/setting-up-snowflake.md
similarity index 84%
rename from website/docs/tutorial/getting-set-up/setting-up-snowflake.md
rename to website/docs/docs/get-started/getting-started/getting-set-up/setting-up-snowflake.md
index 150e6c2b0f0..86ad73d3d47 100644
--- a/website/docs/tutorial/getting-set-up/setting-up-snowflake.md
+++ b/website/docs/docs/get-started/getting-started/getting-set-up/setting-up-snowflake.md
@@ -3,6 +3,8 @@ title: "Set up and connect Snowflake"
id: setting-up-snowflake
description: "Set up Snowflake with sample data and connect to dbt Cloud."
sidebar_label: "Set up and connect Snowflake"
+pagination_prev: docs/get-started/getting-started/set-up-dbt-cloud
+pagination_next: docs/get-started/getting-started/building-your-first-project/build-your-first-models
---
## Introduction
@@ -19,7 +21,7 @@ This guide will walk you through:
## Prerequisites
-The only prerequisites for this tutorial are to have access to an email account for signing up for Snowflake and dbt Cloud.
+The only prerequisites for this guide are to have access to an email account for signing up for Snowflake and dbt Cloud.
## Setting up
@@ -173,7 +175,7 @@ There are two ways to connect dbt Cloud and Snowflake. The first option is Partn
### Option 1: Connect dbt Cloud and Snowflake with partner connect
-1. With your Snowflake account up and running with data, we’re ready to connect it with dbt Cloud. We’re going to use [Snowflake Partner Connect](https://docs.snowflake.com/en/user-guide/ecosystem-partner-connect.html) to set up your dbt Cloud account and project. Using Partner Connect will allow you to create a complete dbt account with your [Snowflake connection](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-your-database#connecting-to-snowflake), [a managed repository](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/cloud-using-a-managed-repository), [environments](https://docs.getdbt.com/docs/guides/managing-environments), and credentials.
+1. With your Snowflake account up and running with data, we’re ready to connect it with dbt Cloud. We’re going to use [Snowflake Partner Connect](https://docs.snowflake.com/en/user-guide/ecosystem-partner-connect.html) to set up your dbt Cloud account and project. Using Partner Connect will allow you to create a complete dbt account with your [Snowflake connection](https://docs.getdbt.com/docs/dbt-cloud/cloud-configuring-dbt-cloud/connecting-your-database#connecting-to-snowflake), [a managed repository](/docs/collaborate/git/managed-repository), [environments](/docs/build/custom-schemas#managing-environments), and credentials.
2. There’s a couple of ways you can access the Partner Connect page depending on if you’re navigating in the classic Snowflake UI or the new UI.
* **Snowflake Classic UI:** If you’re using the classic version of the Snowflake UI, you can click the Partner Connect button in the top bar of your account. From there, click on the dbt tile to open up the connect box.
@@ -218,7 +220,7 @@ There are two ways to connect dbt Cloud and Snowflake. The first option is Partn
+
+2. For the name, write `Snowflake` or another simple title.
2. Enter the following information under Snowflake settings.
- * **Account:** Your account is going to be the url of your Snowflake trial account up to the period before snowflakecomputing.com. So, if this is the full url of my trial account: `oq65696.west-us-2.azure.snowflakecomputing.com` then the account will be: `oq65696.west-us-2.azure`. You can read more about Snowflake account identifiers [here](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html).
+ * **Account:** Find your account by using the Snowflake trial account URL and removing `snowflakecomputing.com`. The order of your account information will vary by Snowflake version. For example, Snowflake's Classic console URL might look like: `oq65696.west-us-2.azure.snowflakecomputing.com`. The AppUI or Snowsight URL might look more like: `snowflakecomputing.com/west-us-2.azure/oq65696`. In both examples, your account will be: `oq65696.west-us-2.azure`. For more information, see "[Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html)" in the Snowflake documentation.
+ * **Role:** Leave blank for now. You can update this to a default Snowflake role in the future. * **Database:** `analytics`. This tells dbt to create new models in the analytics database. * **Warehouse:** `transforming`. This tells dbt to use the transforming warehouse we created earlier. -
+
+
+
++- Have both your production and development environments running dbt version 1.3 or higher
+- Use Snowflake data platform
+- Install the dbt metrics package version
">=1.3.0", "<1.4.0" in your dbt project +- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+- Have both your production and development environments running dbt version 1.2 (latest)
+- Use Snowflake data platform
+- Install the dbt metrics package version
">=0.3.0", "<0.4.0" in your dbt project +- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
_* Available during Public Preview only_ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise) | +| **[Metadata API](/docs/dbt-cloud-apis/metadata-api)** | Accesses metric definitions primarily via integrations and is the source of truth for objects defined in dbt projects (like models, macros, sources, metrics). The Metadata API is updated at the end of every dbt Cloud run. | ❌ | ✅ | ✅ | Proprietary, Cloud (Team & Enterprise | + +
+ +- [How to design and structure dbt metrics: Recommendations for getting started](https://docs.getdbt.com/blog/how-to-design-and-structure-metrics) blog to understand best practices for designing and structuring metrics in your dbt project +- [dbt metrics](docs/build/metrics) for in-depth detail on attributes, filters, how to define and query your metrics and [dbt-metrics package](https://github.com/dbt-labs/dbt_metrics) +- [dbt Semantic Layer quickstart](/docs/use-dbt-semantic-layer/quickstart-semantic-layer) to get started +- [Understanding the components of the dbt Semantic Layer](https://docs.getdbt.com/blog/understanding-the-components-of-the-dbt-semantic-layer) blog post to see further examples + +Review our helpful metrics video below, which explains what metrics are, why they're important and how you can get started: + +
+
+How are you storing my data?
+
+
+The dbt Semantic Layer does not store, or cache, or log your data. On each query to the Semantic Layer, the resulting data passes through dbt Cloud servers where it is never stored, cached, or logged. The data from your data platform gets routed through dbt Cloud servers, to your connecting data tool.
+
+
+Is the dbt Semantic Layer open source?
+
+
+Some components of the dbt Semantic Layer are open source like dbt-core, the dbt_metrics package, and the BSL licensed dbt-server. The dbt Proxy Server (what is actually compiling the dbt code) and the Metadata API are not open source.
+ +During Public Preview, the dbt Semantic Layer is open to all dbt Cloud tiers (Developer, Team, and Enterprise).
+ +
+ +During Public Preview, the dbt Semantic Layer is open to all dbt Cloud tiers (Developer, Team, and Enterprise).
+ +
-
+
- dbt Core users can define metrics in their dbt Core projects and calculate them using macros from the metrics package. To use the dbt Semantic Layer integrations, users will need to have a dbt Cloud account.
- Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API.
- Team and Enterprise accounts will be able to set up the Semantic Layer and Metadata API in the integrated partner tool to import metric definition. +
+
+
+
+ Is there a dbt Semantic Layer discussion hub?
+
+
+Yes absolutely! Join the dbt Slack community and #dbt-cloud-semantic-layer slack channel for all things related to the dbt Semantic Layer.
+
+ diff --git a/website/docs/docs/use-dbt-semantic-layer/quickstart-semantic-layer.md b/website/docs/docs/use-dbt-semantic-layer/quickstart-semantic-layer.md new file mode 100644 index 00000000000..2ed60e32ded --- /dev/null +++ b/website/docs/docs/use-dbt-semantic-layer/quickstart-semantic-layer.md @@ -0,0 +1,321 @@ +--- +title: "Quickstart" +id: quickstart-semantic-layer +description: "Define metrics and set up the dbt Semantic Layer" +sidebar_label: "Quickstart" +--- + +# dbt Semantic Layer quickstart + +
+- Have both your production and development environments running dbt version 1.3 or higher
+- Use Snowflake data platform
+- Install the dbt metrics package version
">=1.3.0", "<1.4.0" in your dbt project +- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+- Have both your production and development environments running dbt version 1.2 (latest)
+- Use Snowflake data platform
+- Install the dbt metrics package version
">=0.3.0", "<0.4.0" in your dbt project +- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but won't be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+
+How are you storing my data?
+
+
+The dbt Semantic Layer does not store, or cache, or log your data. On each query to the Semantic Layer, the resulting data passes through dbt Cloud servers where it is never stored, cached, or logged. The data from your data platform gets routed through dbt Cloud servers, to your connecting data tool.
+
+
+Is the dbt Semantic Layer open source?
+
+
+Some components of the dbt Semantic Layer are open source like dbt-core, the dbt_metrics package, and the BSL licensed dbt-server. The dbt Proxy Server (what is actually compiling the dbt code) and the Metadata API are not open source.
+ +During Public Preview, the dbt Semantic Layer is open to all dbt Cloud tiers (Developer, Team, and Enterprise).
+
+ + +During Public Preview, the dbt Semantic Layer is open to all dbt Cloud tiers (Developer, Team, and Enterprise).
+
-
+
- dbt Core users can define metrics in their dbt Core projects and calculate them using macros from the metrics package. To use the dbt Semantic Layer integrations, you will need to have a dbt Cloud account.
- Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API.
- Team and Enterprise accounts will be able to set up the Semantic Layer and Metadata API in the integrated partner tool to import metric definitions. +
+
+
+ The
+
+The dbt_metrics_calendar_table does not exist or is not authorized?
+
+
+All metrics queries are dependent on either the
+ +Fix: + +
+ dbt_metrics_calendar_table or a custom calendar set in the users dbt_project.yml. If you have not created this model in the database, these queries will fail and you'll most likely see the following error message:
+
+Object DATABASE.SCHEMA.DBT_METRICS_DEFAULT_CALENDAR does not exist or not authorized.+ +Fix: + +
-
+
- If developing locally, run
dbt run --select dbt_metrics_default_calendar - If you are using this in production, make sure that you perform a full
dbt buildordbt run. If you are running specificselectsin your production job, then you will not create this required model.
+
+
+
+
+Ephemeral Models - Object does not exist or is not authorized
+
+
+Metrics cannot be defined on ephemeral models because we reference the underlying table in the query that generates the metric so we need the table/view to exist in the database. If your table/view does not exist in your database, you might see this error message:
+
+
+ +Fix: +
+ Object 'DATABASE.SCHEMA.METRIC_MODEL_TABLE' does not exist or not authorized.+ +Fix: +
-
+
- You will need to materialize the model that the metric is built on as a table/view/incremental. +
+
+ Mismatched Versions - metric type is ‘’
+
+
+If you’re running
+ +Fix: + +
+ dbt_metrics ≥v0.3.2 but have dbt-core version ≥1.3.0, you’ll likely see these error messages:
+
+-
+
- Error message 1:
The metric NAME also references ... but its type is ''. Only metrics of type expression can reference other metrics.
+ - Error message 2:
Unknown aggregation style: > in macro default__gen_primary_metric_aggregate (macros/sql_gen/gen_primary_metric_aggregate.sql)
+
type property of the metric spec in dbt-core v1.3.0. The new name is calculation_method and the package reflects that new name, so it isn’t finding any type when we try and run outdated code on it.+ +Fix: + +
-
+
- Upgrade your dbt_metrics package to v1.3.0 +
+ + +## Next steps + +Are you ready to define your own metrics and bring consistency to data consumers? Review the following documents to understand how to structure, define, and query metrics, and set up the dbt Semantic Layer: + +- [How to design and structure dbt metrics: Recommendations for getting started](https://docs.getdbt.com/blog/how-to-design-and-structure-metrics) to understand best practices for designing and structuring metrics in your dbt project +- [dbt metrics](/docs/build/metrics) for in-depth detail on attributes, properties, filters, and how to define and query metrics +- [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-semantic-layer) to learn about the dbt Semantic Layer +- [Understanding the components of the dbt Semantic Layer](https://docs.getdbt.com/blog/understanding-the-components-of-the-dbt-semantic-layer) blog post to see further examples +- [Integrated partner tools](https://www.getdbt.com/product/semantic-layer-integrations) for info on the different integration partners and their documentation +- [dbt Server repo](https://github.com/dbt-labs/dbt-server), which is a persisted HTTP server that wraps dbt core to handle RESTful API requests for dbt operations. diff --git a/website/docs/docs/use-dbt-semantic-layer/set-dbt-semantic-layer.md b/website/docs/docs/use-dbt-semantic-layer/set-dbt-semantic-layer.md new file mode 100644 index 00000000000..90fabea0bca --- /dev/null +++ b/website/docs/docs/use-dbt-semantic-layer/set-dbt-semantic-layer.md @@ -0,0 +1,54 @@ +--- +title: "Set up the dbt Semantic Layer" +id: setup-dbt-semantic-layer +description: "You can set up the dbt Semantic Layer in dbt Cloud." +sidebar_label: "Set up the dbt Semantic Layer" +--- + +With the dbt Semantic Layer, you'll be able to centrally define business metrics, reduce code duplication and inconsistency, create self-service in downstream tools, and more. Configure the dbt Semantic Layer in dbt Cloud to connect with your integrated partner tool. + +## Prerequisites + +Before you set up the dbt Semantic Layer, make sure you meet the following: + +
+- Have both your production and development environments running dbt version 1.3 or higher
+- Use Snowflake data platform
+- Install the dbt metrics package version
">=1.3.0", "<1.4.0" in your dbt project +- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+- Have both your production and development environments running dbt version 1.2 (latest)
+- Use Snowflake data platform
+- Install the dbt metrics package version
">=0.3.0", "<0.4.0" in your dbt project +- Set up the Metadata API in the integrated tool to import metric definitions + * Developer accounts will be able to query the Proxy Server using SQL, but will not be able to browse pre-populated dbt metrics in external tools, which requires access to the Metadata API
+- Recommended - Review the dbt metrics page and Understanding the components of the dbt Semantic Layer blog
+ +
+ + +## Related docs + +- [Integrated partner tools](https://www.getdbt.com/product/semantic-layer-integrations) for info on the different integration partners and their documentation +- [Product architecture](/docs/use-dbt-semantic-layer/dbt-semantic-layer#product-architecture) page for more information on plan availability +- [dbt metrics](/docs/build/metrics) for in-depth detail on attributes, properties, filters, and how to define and query metrics +- [dbt Server repo](https://github.com/dbt-labs/dbt-server), which is a persisted HTTP server that wraps dbt core to handle RESTful API requests for dbt operations diff --git a/website/docs/faqs/Accounts/change-billing.md b/website/docs/faqs/Accounts/change-billing.md new file mode 100644 index 00000000000..65e47cc9c4a --- /dev/null +++ b/website/docs/faqs/Accounts/change-billing.md @@ -0,0 +1,11 @@ +--- +title: How can I update my billing information? +description: "Instructions on how to update your billing info" +sidebar_label: 'How to update billing' +id: change-billing +--- + + +If you want to change your account's credit card details, select the gear menu in the upper right corner of dbt Cloud. Go to Account Settings → Billing → Payment Information. In the upper right corner of Payment Information, click **Edit** to enter the new credit card details. Only the _account owner_ can make this change. + +To change your billing name or location address, send our Support team a message at support@getdbt.com with the newly updated information, and we can make that change for you! diff --git a/website/docs/faqs/configurable-snapshot-path.md b/website/docs/faqs/Accounts/configurable-snapshot-path.md similarity index 76% rename from website/docs/faqs/configurable-snapshot-path.md rename to website/docs/faqs/Accounts/configurable-snapshot-path.md index 66f084ee775..45e51f20dbf 100644 --- a/website/docs/faqs/configurable-snapshot-path.md +++ b/website/docs/faqs/Accounts/configurable-snapshot-path.md @@ -1,5 +1,8 @@ --- title: Can I store my snapshots in a directory other than the `snapshot` directory in my project? +description: "You can see how to change snapshots directory in your project" +sidebar_label: 'Store snapshot in other directory' +id: configurable-snapshot-path --- By default, dbt expects your snapshot files to be located in the `snapshots` subdirectory of your project. diff --git a/website/docs/faqs/Accounts/dbt-specific-jinja.md b/website/docs/faqs/Accounts/dbt-specific-jinja.md new file mode 100644 index 00000000000..e2623491007 --- /dev/null +++ b/website/docs/faqs/Accounts/dbt-specific-jinja.md @@ -0,0 +1,8 @@ +--- +title: What parts of Jinja are dbt-specific? +description: "You can check which Jinja functions are dbt-specific" +sidebar_label: 'dbt-specific Jinja' +id: dbt-specific-jinja +--- + +There are certain expressions that are specific to dbt — these are documented in the [Jinja function reference](/reference/dbt-jinja-functions) section of these docs. Further, docs blocks, snapshots, and
| calendar_date + | calendar_date | -ad_id + | ad_id | -impressions + | impressions | -spend + | spend | -clicks + | clicks | -conversions + | conversions | ||
| unique_id + | unique_id | -calendar_date + | calendar_date | -ad_id + | ad_id | -impressions + | impressions | -spend + | spend | -clicks + | clicks | -conversions + | conversions |
+ ) : isError || !topics?.length > 0 ? (
+