Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt Core and DuckDB Quickstart #4087

Closed
wants to merge 88 commits into from
Closed
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
c5b32a8
dbt Core and DuckDB Quickstart
matthewshaver Sep 18, 2023
0bc1317
Adding updated sections related to duckcli
matthewshaver Sep 18, 2023
5f63efb
Merge branch 'current' into duckdb-qs
matthewshaver Sep 18, 2023
6fc13b2
Merge branch 'duckdb-qs' of https://github.com/dbt-labs/docs.getdbt.c…
matthewshaver Sep 18, 2023
08dd3af
Update duckdb-core-qs.md
matthewshaver Sep 18, 2023
db6bec1
Merge branch 'current' into duckdb-qs
matthewshaver Oct 18, 2023
00a78d6
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
593b63e
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
447a9c5
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
0bb17de
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
97c7395
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
34e514e
Apply suggestions from code review
matthewshaver Oct 19, 2023
558cfa9
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
40f4f7e
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
8575e51
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
5c1b623
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
2fb3768
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
9aef05e
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
3ec7682
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
acec982
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
52358c3
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
370eb43
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
35f1ef8
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
d3e634a
Update duckdb-core-qs.md
matthewshaver Oct 19, 2023
25752f7
Update website/docs/quickstarts/duckdb-core-qs.md
matthewshaver Oct 19, 2023
fb9bc4b
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
55ee64e
This branch was auto-updated!
github-actions[bot] Oct 19, 2023
f098d31
This branch was auto-updated!
github-actions[bot] Oct 20, 2023
bd369d6
This branch was auto-updated!
github-actions[bot] Oct 21, 2023
4c6faf7
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
471e3c3
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
bc9f52c
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
d956b83
This branch was auto-updated!
github-actions[bot] Oct 23, 2023
3ed273a
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
76ab6dc
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
b9523af
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
50ac845
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
889efe8
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
23a4bbb
This branch was auto-updated!
github-actions[bot] Oct 24, 2023
3db14f7
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
f527811
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
3de866e
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
15ef5e7
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
9344955
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
e0922e4
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
2947fbf
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
5bf0235
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
e541d1b
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
5a9282b
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
f48e8c2
This branch was auto-updated!
github-actions[bot] Oct 25, 2023
ab3b239
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
ead948f
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
9292449
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
04ab89a
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
016bd16
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
ce8b677
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
965fd9a
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
291c6e9
This branch was auto-updated!
github-actions[bot] Oct 26, 2023
6e94df6
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
89ce1e2
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
c8dc512
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
0b78112
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
26d9a5b
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
c547ef4
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
46902d8
This branch was auto-updated!
github-actions[bot] Oct 27, 2023
33900eb
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
b3fcb10
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
a40283e
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
4500cb9
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
6fd7c72
This branch was auto-updated!
github-actions[bot] Oct 30, 2023
a6a90ee
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
864d56c
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
7c59d26
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
ffd1701
This branch was auto-updated!
github-actions[bot] Oct 31, 2023
5fcd494
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
5c2b12a
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
bbca294
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
dc05baf
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
6b2effd
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
fc46eeb
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
f7b4f3c
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
aae108c
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
937e261
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
de00f64
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
0a255d0
This branch was auto-updated!
github-actions[bot] Nov 1, 2023
05c9b6f
This branch was auto-updated!
github-actions[bot] Nov 2, 2023
3e05c18
This branch was auto-updated!
github-actions[bot] Nov 2, 2023
37d83f1
This branch was auto-updated!
github-actions[bot] Nov 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 140 additions & 0 deletions website/docs/quickstarts/duckdb-core-qs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
---
title: "Quickstart for dbt Core using DuckDB"
id: duckdb
description: "Install dbt Core and DuckDB to begin testing within minutes"
platform: 'dbt-core'
icon: 'fa-light fa-square-terminal'
hide_table_of_contents: true
---

## Introduction

In this quickstart guide, you’ll learn how to install dbt Core and DuckDB, and import sample data from the fictional e-commerce Jaffle Shop data set within minutes. DuckDB is an in-process SQL OLAP database management system optimized for data analytics. Installing dbt Core and DuckDB together allows you to test all of the available features and functionality dbt has to offer locally on your computer (no data warehouse required), and the Jaffle Shop provides a pre-built dbt project ready to run out of the box.
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

## Prerequisites

* To use the dbt CLI, you must know some command line basics. In particular, you should understand `cd`, `ls` and `pwd` to navigate through the directory structure of your computer.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
* A Windows, Mac, or Linux computer with sufficient storage. The Jaffle Shop sample data set is small and lightweight, taking up less than 200MB.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
* [Python 3.5](https://www.python.org/downloads/) or higher.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
* [Create a GitHub account](https://github.com/join) if you don't already have one. [Set up Git](https://docs.github.com/en/get-started/quickstart/set-up-git) on your computer.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
* (Optional) [GitHub Codespace](https://docs.github.com/en/codespaces/managing-codespaces-for-your-organization/enabling-or-disabling-github-codespaces-for-your-organization) enabled for your account.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

## Installation instructions

Select the following format that applies to your environment. Copy and paste the commands (you can copy them all at once or line-by-line) into your command line interface(CLI) and execute them. The syntax may vary slightly for each, but the result of the commands is the same. They will:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

- Clone the [jaffle_shop_duckdb github repo](https://github.com/dbt-labs/jaffle_shop_duckdb) to your local machine.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- Change the directory to the Jaffle Shop.
- Create a Python virtual environment.
- Install and update pip.
- Install dbt Core and dependencies.
- Install DuckDB and dependencies.
- Build the Jaffle Shop project in dbt Core.
- Generate (`dbt docs generate`) and serve (`dbt docs serve`) dbt docs to your browser (localhost).

For the complete list of dependencies that will be installed, read the [**requirements.txt**](https://github.com/dbt-labs/jaffle_shop_duckdb/blob/duckdb/requirements.txt) file in the jaffle_shop_duckdb Github repo.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

<Tabs>
<TabItem value="Bash" label="bash/zsh (Mac)">

```shell
git clone https://github.com/dbt-labs/jaffle_shop_duckdb.git
cd jaffle_shop_duckdb
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
source venv/bin/activate
dbt build
dbt docs generate
dbt docs serve
```

</TabItem>
<TabItem value="csh" label="csh/tcsh">

```shell
git clone https://github.com/dbt-labs/jaffle_shop_duckdb.git
cd jaffle_shop_duckdb
python3 -m venv venv
source venv/bin/activate.csh
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
source venv/bin/activate.csh
dbt build
dbt docs generate
dbt docs serve
```

</TabItem>
<TabItem value="powershell" label="Windows Powershell">

```shell
git clone https://github.com/dbt-labs/jaffle_shop_duckdb.git
cd jaffle_shop_duckdb
python -m venv venv
venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
venv\Scripts\Activate.ps1
dbt build
dbt docs generate
dbt docs serve
```

</TabItem>
<TabItem value="codespace" label="GitHub Codespaces">

#### Steps:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

1. Ensure you have [Codespaces](https://github.com/features/codespaces) enabled for your GitHub organization or turned on as a beta feature if you're an individual user.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
2. Open the [jaffle_shop_duckdb](https://github.com/dbt-labs/jaffle_shop_duckdb) repo in your browser.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
3. Click the green **Code** button near the top right of the repo's homepage (you may already be on it).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
4. Rather than cloning the repo like you normally would, click the **Codespaces** tab of the pop-out and then click **Create codespace on `duckdb`**.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
<div style={{maxWidth: '400px'}}>
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
<Lightbox src="/img/codespace-quickstart/open_in_codespaces.png" title="Open in Codespaces" />
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
</div>
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

5. Wait for Codespaces to boot (this may take a few minutes).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
6. Select whether you'd like to use the Web IDE or open the codespace in your local environment.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
7. When the Codespaces opens, a **Task** pane will show up and call `dbt build` to show you how it's done.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
8. (Optional) Install the recommended extensions (like **dbt Power User extension**).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
9. Open up a new terminal and type: `dbt build`
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

</TabItem>
</Tabs>

## Query the data

The Jaffle Shop project will seed the data set to DuckDB, which can then be manually queried from the duckcli tool.
Copy link
Contributor

@nghi-ly nghi-ly Oct 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The Jaffle Shop project will seed the data set to DuckDB, which can then be manually queried from the duckcli tool.
The Jaffle Shop project seeds the data set to DuckDB, which can then be manually queried from the DuckCLI tool.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duckcli is a product name, so shouldn't be in code blocks.

Copy link
Contributor

@nghi-ly nghi-ly Oct 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah gotcha! updated it so it refers to the product name versus the command name (one uses to execute it)

Screenshot 2023-10-19 at 2 15 08 PM

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

Launch the DuckDB command-line interface (CLI):

```shell
duckcli jaffle_shop.duckdb
```

Run a query at the prompt and exit:

```
select * from customers where customer_id = 42;
exit;
```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the above examples are not rendering properly. i think if you remove the leading whitespace it'll fix this issue.

## Browsing the data
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

There are a few options available to browse the data within DuckDB:

- [duckcli](https://pypi.org/project/duckcli/)
- [DuckDB CLI](https://duckdb.org/docs/installation/?environment=cli)
- [DBeaver SQL IDE for DuckDB](https://duckdb.org/docs/guides/sql_editors/dbeaver)

## Next steps

Now that you've got dbt Core, DuckDB, and the Jaffle Shop data up and running, it's time to explore dbt's capabilities. Read through the following material to get a better understanding of dbt projects and commands:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

* [About projects](/docs/build/projects) will guide you through the structure of a dbt project and its components.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
* The [dbt Command reference](/reference/dbt-commands) will explain the various commands available and what they do.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
* The [Courses](https://courses.getdbt.com/collections) offer a variety of beginner, intermediate, and advanced learning modules designed to help you become a dbt expert.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
* Once you see the potential of dbt and what it can do for your organization, sign up for a free trial of [dbt Cloud](https://www.getdbt.com/signup). It's the fastest and easiest way to deploy dbt today!
* Check out our other [Quickstart guides](/quickstarts) to begin integrating into your existing data warehouse.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading