Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt Core and DuckDB Quickstart #4087

Closed
wants to merge 88 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
c5b32a8
dbt Core and DuckDB Quickstart
matthewshaver Sep 18, 2023
0bc1317
Adding updated sections related to duckcli
matthewshaver Sep 18, 2023
5f63efb
Merge branch 'current' into duckdb-qs
matthewshaver Sep 18, 2023
6fc13b2
Merge branch 'duckdb-qs' of https://github.com/dbt-labs/docs.getdbt.c…
matthewshaver Sep 18, 2023
08dd3af
Update duckdb-core-qs.md
matthewshaver Sep 18, 2023
db6bec1
Merge branch 'current' into duckdb-qs
matthewshaver Oct 18, 2023
00a78d6
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
593b63e
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
447a9c5
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
0bb17de
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
97c7395
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
34e514e
Apply suggestions from code review
matthewshaver Oct 19, 2023
558cfa9
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
40f4f7e
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
8575e51
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
5c1b623
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
2fb3768
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
9aef05e
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
3ec7682
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
acec982
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
52358c3
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
370eb43
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
35f1ef8
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
d3e634a
Update duckdb-core-qs.md
matthewshaver Oct 19, 2023
25752f7
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
fb9bc4b
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
55ee64e
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
f098d31
This branch was auto-updated!
github-actions[bot] Oct 20, 2023
bd369d6
This branch was auto-updated!
github-actions[bot] Oct 21, 2023
4c6faf7
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
471e3c3
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
bc9f52c
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
d956b83
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
3ed273a
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
76ab6dc
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
b9523af
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
50ac845
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
889efe8
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
23a4bbb
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
3db14f7
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
f527811
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
3de866e
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
15ef5e7
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
9344955
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
e0922e4
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
2947fbf
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
5bf0235
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
e541d1b
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
5a9282b
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
f48e8c2
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
ab3b239
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
ead948f
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
9292449
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
04ab89a
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
016bd16
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
ce8b677
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
965fd9a
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
291c6e9
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
6e94df6
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
89ce1e2
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
c8dc512
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
0b78112
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
26d9a5b
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
c547ef4
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
46902d8
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
33900eb
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
b3fcb10
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
a40283e
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
4500cb9
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
6fd7c72
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
a6a90ee
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
864d56c
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
7c59d26
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
ffd1701
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
5fcd494
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
5c2b12a
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
bbca294
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
dc05baf
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
6b2effd
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
fc46eeb
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
f7b4f3c
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
aae108c
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
937e261
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
de00f64
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
0a255d0
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
05c9b6f
This branch was auto-updated!
github-actions[bot] Nov 2, 2023
3e05c18
This branch was auto-updated!
github-actions[bot] Nov 2, 2023
37d83f1
This branch was auto-updated!
github-actions[bot] Nov 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 144 additions & 0 deletions website/docs/quickstarts/duckdb-core-qs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
---
title: "Quickstart for dbt Core using DuckDB"
id: duckdb
description: "Install dbt Core and DuckDB to begin testing within minutes"
platform: 'dbt-core'
icon: 'fa-light fa-square-terminal'
hide_table_of_contents: true
---

## Introduction

In this quickstart guide, you’ll learn how to install dbt Core and DuckDB, and import sample data from the fictional e-commerce Jaffle Shop data set within minutes. DuckDB is an in-process SQL OLAP database management system optimized for data analytics. Installing dbt and DuckDB together enables you to test all the available features and functionality dbt has to offer locally on your computer (no data warehouse required), and the Jaffle Shop Git repository provides a prebuilt dbt project that's ready to run out of the box.

## Prerequisites

* You must know some command line basics to use dbt Core. In particular, you should be familiar with `cd`, `ls` and `pwd` so you can navigate through the directory structure of your computer.
* You have a Windows, Mac, or Linux computer with sufficient storage. The Jaffle Shop sample data set is small and lightweight, taking up less than 200 MB.
* You must have [Python 3.5](https://www.python.org/downloads/) or newer.
* You must have a [GitHub account](https://github.com/join). For details, refer to [Set up Git](https://docs.github.com/en/get-started/quickstart/set-up-git) in the GitHub documentation.
* (Optional) [GitHub Codespace](https://docs.github.com/en/codespaces/managing-codespaces-for-your-organization/enabling-or-disabling-github-codespaces-for-your-organization) enabled for your account.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

## Installation instructions

Select a format below that applies to your environment. Copy and paste the commands (you can copy them all at once or line-by-line) into your command line interface (CLI) and execute them. The syntax varies slightly for each, but the result of the commands is the same. They will:

- Clone the [jaffle_shop_duckdb GitHub repo](https://github.com/dbt-labs/jaffle_shop_duckdb) to your local machine.
- Change the directory to the Jaffle Shop.
- Create a Python virtual environment.
- Install and update pip.
- Install dbt Core and dependencies.
- Install DuckDB and dependencies.
- Build the Jaffle Shop project in dbt Core.
- Generate (`dbt docs generate`) and serve (`dbt docs serve`) dbt docs to your browser (localhost).

For the complete list of installed dependencies, refer to the [requirements.txt](https://github.com/dbt-labs/jaffle_shop_duckdb/blob/duckdb/requirements.txt) file in the jaffle_shop_duckdb GitHub repo.

<Tabs>
<TabItem value="Bash" label="bash/zsh (Mac)">

```shell
git clone https://github.com/dbt-labs/jaffle_shop_duckdb.git
cd jaffle_shop_duckdb
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
source venv/bin/activate
dbt build
dbt docs generate
dbt docs serve
```

</TabItem>
<TabItem value="csh" label="csh/tcsh">

```shell
git clone https://github.com/dbt-labs/jaffle_shop_duckdb.git
cd jaffle_shop_duckdb
python3 -m venv venv
source venv/bin/activate.csh
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
source venv/bin/activate.csh
dbt build
dbt docs generate
dbt docs serve
```

</TabItem>
<TabItem value="powershell" label="Windows Powershell">

```shell
git clone https://github.com/dbt-labs/jaffle_shop_duckdb.git
cd jaffle_shop_duckdb
python -m venv venv
venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
venv\Scripts\Activate.ps1
dbt build
dbt docs generate
dbt docs serve
```

</TabItem>
<TabItem value="codespace" label="GitHub Codespaces">

#### Steps:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

1. Ensure you have [Codespaces](https://github.com/features/codespaces) enabled for your GitHub organization or turned on as a beta feature if you're an individual user.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
2. Open the [jaffle_shop_duckdb](https://github.com/dbt-labs/jaffle_shop_duckdb) repo in your browser.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
3. Click the green **Code** button near the top right of the repo's homepage (you may already be on it).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
4. Rather than cloning the repo like you normally would, click the **Codespaces** tab of the pop-out and then click **Create codespace on `duckdb`**.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
<div style={{maxWidth: '400px'}}>
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
<Lightbox src="/img/codespace-quickstart/open_in_codespaces.png" title="Open in Codespaces" />
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
</div>
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

5. Wait for Codespaces to boot (this may take a few minutes).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
6. Select whether you'd like to use the Web IDE or open the codespace in your local environment.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
7. When the Codespaces opens, a **Task** pane will show up and call `dbt build` to show you how it's done.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
8. (Optional) Install the recommended extensions (like **dbt Power User extension**).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
9. Open up a new terminal and type: `dbt build`
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

</TabItem>
</Tabs>

## Query the data

The Jaffle Shop project seeds the data set to DuckDB, which can then be manually queried from the duckcli tool.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The Jaffle Shop project seeds the data set to DuckDB, which can then be manually queried from the duckcli tool.
The Jaffle Shop project seeds the data set to DuckDB, which can then be manually queried from the DuckCLI tool.

since you're referring to the product name (good point!), sugg match its casing like this:

Screenshot 2023-10-19 at 2 15 08 PM


Launch the DuckDB command-line interface (CLI):

```shell

duckcli jaffle_shop.duckdb

```

Run a query at the prompt and exit:

```sql

select * from customers where customer_id = 42;
exit;

```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the above examples are not rendering properly. i think if you remove the leading whitespace it'll fix this issue.

## Browse the data

There are a few options available to browse the data within DuckDB:

- [duckcli](https://pypi.org/project/duckcli/)
- [DuckDB CLI](https://duckdb.org/docs/installation/?environment=cli)
- [DBeaver SQL IDE for DuckDB](https://duckdb.org/docs/guides/sql_editors/dbeaver)

## Next steps

Now that you've got dbt Core, DuckDB, and the Jaffle Shop data up and running, you can explore dbt's capabilities. Refer to these materials to get a better understanding of dbt projects and commands:

* The [About projects](/docs/build/projects) page guides you through the structure of a dbt project and its components.
* The [dbt command reference](/reference/dbt-commands) explains the various commands available and what they do.
* The [dbt Labs courses](https://courses.getdbt.com/collections) offer a variety of beginner, intermediate, and advanced learning modules designed to help you become a dbt expert.
* Once you see the potential of dbt and what it can do for your organization, sign up for a free trial of [dbt Cloud](https://www.getdbt.com/signup). It's the fastest and easiest way to deploy dbt today!
* Check out the other [quickstart guides](/quickstarts) to begin integrating into your existing data warehouse.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure where to put this so adding it here (apologies for being "out of place"!)

you should also add a new tile for this new qs. it's in the website/vercel.json file.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading