Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure private link #5929

Merged
merged 16 commits into from
Aug 13, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 20 additions & 3 deletions website/docs/docs/cloud/secure/databricks-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps will walk you through the setup of a Databricks AWS PrivateLink endpoint in the dbt Cloud multi-tenant environment.
The following steps will walk you through the setup of a Databricks AWS PrivateLink or Azure Private Link endpoint in the dbt Cloud multi-tenant environment.

## Configure PrivateLink
## Configure AWS PrivateLink

1. Locate your [Databricks instance name](https://docs.databricks.com/en/workspace/workspace-details.html#workspace-instance-names-urls-and-ids)
- Example: `cust-success.cloud.databricks.com`
2. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
```
Subject: New Multi-Tenant PrivateLink Request
Subject: New AWS Multi-Tenant PrivateLink Request
- Type: Databricks
- Databricks instance name:
- Databricks cluster AWS Region (e.g., us-east-1, eu-west-2):
Expand All @@ -41,6 +41,23 @@ If using an existing Databricks workspace, all workloads running in the workspac

:::

## Configure Azure Private Link

1. Navigate to your Azure Databricks workspace.
The path is: `/subscriptions/<subscription_uuid>/resourceGroups/<resource_group_name>/providers/Microsoft.Databricks/workspaces/<workspace_name>`.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
2. From the workspace overview, click **JSON view**.
3. Copy the value in the `resource_id` field.
4. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
```
Subject: New Azure Multi-Tenant Private Link Request
- Type: Databricks
- Databricks instance name:
- Databricks Azure resource ID:
- dbt Cloud multi-tenant environment: EMEA
```
5. Once Support confirms the resources are available in the Azure portal, navigate to the Azure Databricks Workspace and go to **Networking** --> **Private Endpoint Connections** --> highlight the `dbt` named option, and select **Approve**.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved


## Create Connection in dbt Cloud

Once you've completed the setup in the Databricks environment, you will be able to configure a private endpoint in dbt Cloud:
Expand Down
42 changes: 35 additions & 7 deletions website/docs/docs/cloud/secure/snowflake-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps walk you through the setup of a Snowflake AWS PrivateLink and Azure Private Link endpoint in the dbt Cloud multi-tenant environment.
The following steps walk you through the setup of a Snowflake AWS PrivateLink or Azure Private Link endpoint in a dbt Cloud multi-tenant environment.

:::note Snowflake SSO with PrivateLink
Users connecting to Snowflake using SSO over a PrivateLink connection from dbt Cloud will also require access to a PrivateLink endpoint from their local workstation.
Expand All @@ -21,10 +21,11 @@ Users connecting to Snowflake using SSO over a PrivateLink connection from dbt C

## Configure PrivateLink

1. Open a Support case with Snowflake to allow access from the dbt Cloud AWS account
- Snowflake prefers that the account owner opens the Support case directly, rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors)
1. Open a Support case with Snowflake to allow access from the dbt Cloud AWS or Entra ID account.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- Snowflake prefers that the account owner opens the Support case directly, rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- Provide them with your dbt Cloud account ID along with any other information requested in the article.
- AWS account ID: `346425330055` - _NOTE: This account ID only applies to dbt Cloud Multi-Tenant environments. For Virtual Private/Single-Tenant account IDs please contact [Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support)._
- **AWS account ID**: `346425330055` &mdash; _NOTE: This account ID only applies to AWS dbt Cloud multi-tenant environments. For AWS Virtual Private/Single-Tenant account IDs please contact [Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support)._
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- Azure private endpoint resource ID: Please contact [Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support).
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
- You will need to have `ACCOUNTADMIN` access to the Snowflake instance to submit a Support request.

<Lightbox src="/img/docs/dbt-cloud/snowflakeprivatelink1.png" title="Open snowflake case"/>
Expand All @@ -34,11 +35,13 @@ Users connecting to Snowflake using SSO over a PrivateLink connection from dbt C
3. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```
Subject: New Multi-Tenant PrivateLink Request
Subject: New Multi-Tenant (Azure or AWS) PrivateLink Request
- Type: Snowflake
- SYSTEM$GET_PRIVATELINK_CONFIG output:
- *Use privatelink-account-url or regionless-privatelink-account-url?:
- dbt Cloud multi-tenant environment (US, EMEA, AU):
- dbt Cloud multi-tenant environment
- AWS: US, EMEA, or AU
- Azure: EMEA only
```
_*By default dbt Cloud will be configured to use `privatelink-account-url` from the provided [SYSTEM$GET_PRIVATELINK_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) as the PrivateLink endpoint. Upon request, `regionless-privatelink-account-url` can be used instead._

Expand All @@ -57,6 +60,27 @@ Once dbt Cloud support completes the configuration, you can start creating new c
4. Configure the remaining data platform details.
5. Test your connection and save it.

## Enable the connection in Snowflake

To complete the setup, follow the remaining steps from the Snowflake setup guides. The instructions vary based on the platform:

- [Snowflake AWS PrivateLink](https://docs.snowflake.com/en/user-guide/admin-security-privatelink)
- [Snowflake Azure Private Link](https://docs.snowflake.com/en/user-guide/privatelink-azure)

There are some nuances for each connection, but in each case, you will need a Snowflake administrator. As the Snowflake administrator, call the `SYSTEM$AUTHORIZE_STAGE_PRIVATELINK_ACCESS` function using the privateEndpointResourceID value as the function argument. This step authorizes access to the Snowflake internal stage through the private endpoint.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```sql

USE ROLE ACCOUNTADMIN;

-- AWS PrivateLink
SELECT SYSTEMS$AUTHORIZE_STATE_PRIVATELINK_ACCESS ( `AWS VPC ID` );

-- Azure Private Link
SELECT SYSTEMS$AUTHORIZE_STATE_PRIVATELINK_ACCESS ( `AZURE PRIVATED ENDPOINT RESOURCE ID` );
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

```

## Configuring Network Policies
If your organization uses [Snowflake Network Policies](https://docs.snowflake.com/en/user-guide/network-policies) to restrict access to your Snowflake account, you will need to add a network rule for dbt Cloud.

Expand Down Expand Up @@ -84,19 +108,23 @@ Open the Snowflake UI and take the following steps:
<Lightbox src="/img/docs/dbt-cloud/snowflakeprivatelink3.png" title="Update Network Policy"/>

### Using SQL

For quick and automated setup of network rules via SQL in Snowflake, the following commands allow you to create and configure access rules for dbt Cloud. These SQL examples demonstrate how to add a network rule and update your network policy accordingly.

1. Create a new network rule with the following SQL:
```sql

CREATE NETWORK RULE allow_dbt_cloud_access
MODE = INGRESS
TYPE = AWSVPCEID
VALUE_LIST = ('<VPCE_ID>'); -- Replace '<VPCE_ID>' with the actual ID provided

```

2. Add the rule to a network policy with the following SQL:
```sql

ALTER NETWORK POLICY <network_policy_name>
ADD ALLOWED_NETWORK_RULE_LIST =('allow_dbt_cloud_access');
```

```
Loading