Skip to content

Commit

Permalink
Azure private link
Browse files Browse the repository at this point in the history
  • Loading branch information
matthewshaver committed Aug 13, 2024
1 parent 0b33165 commit dd8c09c
Show file tree
Hide file tree
Showing 2 changed files with 55 additions and 10 deletions.
23 changes: 20 additions & 3 deletions website/docs/docs/cloud/secure/databricks-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps will walk you through the setup of a Databricks AWS PrivateLink endpoint in the dbt Cloud multi-tenant environment.
The following steps will walk you through the setup of a Databricks AWS PrivateLink or Azure Private Link endpoint in the dbt Cloud multi-tenant environment.

## Configure PrivateLink
## Configure AWS PrivateLink

1. Locate your [Databricks instance name](https://docs.databricks.com/en/workspace/workspace-details.html#workspace-instance-names-urls-and-ids)
- Example: `cust-success.cloud.databricks.com`
2. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
```
Subject: New Multi-Tenant PrivateLink Request
Subject: New AWS Multi-Tenant PrivateLink Request
- Type: Databricks
- Databricks instance name:
- Databricks cluster AWS Region (e.g., us-east-1, eu-west-2):
Expand All @@ -41,6 +41,23 @@ If using an existing Databricks workspace, all workloads running in the workspac
:::

## Configure Azure Private Link

1. Navigate to your Azure Databricks workspace.
The path is: `/subscriptions/<subscription_uuid>/resourceGroups/<resource_group_name>/providers/Microsoft.Databricks/workspaces/<workspace_name>`.
2. From the workspace overview, click **JSON view**.
3. Copy the value in the `resource_id` field.
4. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):
```
Subject: New Azure Multi-Tenant Private Link Request
- Type: Databricks
- Databricks instance name:
- Databricks Azure resource ID:
- dbt Cloud multi-tenant environment: EMEA
```
5. Once Support confirms the resources are available in the Azure portal, navigate to the Azure Databricks Workspace and go to **Networking** --> **Private Endpoint Connections** --> highlight the `dbt` named option, and select **Approve**.
## Create Connection in dbt Cloud
Once you've completed the setup in the Databricks environment, you will be able to configure a private endpoint in dbt Cloud:
Expand Down
42 changes: 35 additions & 7 deletions website/docs/docs/cloud/secure/snowflake-privatelink.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import SetUpPages from '/snippets/_available-tiers-privatelink.md';

<SetUpPages features={'/snippets/_available-tiers-privatelink.md'}/>

The following steps walk you through the setup of a Snowflake AWS PrivateLink and Azure Private Link endpoint in the dbt Cloud multi-tenant environment.
The following steps walk you through the setup of a Snowflake AWS PrivateLink or Azure Private Link endpoint in a dbt Cloud multi-tenant environment.

:::note Snowflake SSO with PrivateLink
Users connecting to Snowflake using SSO over a PrivateLink connection from dbt Cloud will also require access to a PrivateLink endpoint from their local workstation.
Expand All @@ -21,10 +21,11 @@ Users connecting to Snowflake using SSO over a PrivateLink connection from dbt C

## Configure PrivateLink

1. Open a Support case with Snowflake to allow access from the dbt Cloud AWS account
- Snowflake prefers that the account owner opens the Support case directly, rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors)
1. Open a Support case with Snowflake to allow access from the dbt Cloud AWS or Entra ID account.
- Snowflake prefers that the account owner opens the Support case directly, rather than dbt Labs acting on their behalf. For more information, refer to [Snowflake's knowledge base article](https://community.snowflake.com/s/article/HowtosetupPrivatelinktoSnowflakefromCloudServiceVendors).
- Provide them with your dbt Cloud account ID along with any other information requested in the article.
- AWS account ID: `346425330055` - _NOTE: This account ID only applies to dbt Cloud Multi-Tenant environments. For Virtual Private/Single-Tenant account IDs please contact [Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support)._
- **AWS account ID**: `346425330055` &mdash; _NOTE: This account ID only applies to AWS dbt Cloud multi-tenant environments. For AWS Virtual Private/Single-Tenant account IDs please contact [Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support)._
- Azure private endpoint resource ID: Please contact [Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support).
- You will need to have `ACCOUNTADMIN` access to the Snowflake instance to submit a Support request.

<Lightbox src="/img/docs/dbt-cloud/snowflakeprivatelink1.png" title="Open snowflake case"/>
Expand All @@ -34,11 +35,13 @@ Users connecting to Snowflake using SSO over a PrivateLink connection from dbt C
3. Add the required information to the template below, and submit your request to [dbt Support](https://docs.getdbt.com/community/resources/getting-help#dbt-cloud-support):

```
Subject: New Multi-Tenant PrivateLink Request
Subject: New Multi-Tenant (Azure or AWS) PrivateLink Request
- Type: Snowflake
- SYSTEM$GET_PRIVATELINK_CONFIG output:
- *Use privatelink-account-url or regionless-privatelink-account-url?:
- dbt Cloud multi-tenant environment (US, EMEA, AU):
- dbt Cloud multi-tenant environment
- AWS: US, EMEA, or AU
- Azure: EMEA only
```
_*By default dbt Cloud will be configured to use `privatelink-account-url` from the provided [SYSTEM$GET_PRIVATELINK_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config.html) as the PrivateLink endpoint. Upon request, `regionless-privatelink-account-url` can be used instead._

Expand All @@ -57,6 +60,27 @@ Once dbt Cloud support completes the configuration, you can start creating new c
4. Configure the remaining data platform details.
5. Test your connection and save it.

## Enable the connection in Snowflake

To complete the setup, follow the remaining steps from the Snowflake setup guides. The instructions vary based on the platform:

- [Snowflake AWS PrivateLink](https://docs.snowflake.com/en/user-guide/admin-security-privatelink)
- [Snowflake Azure Private Link](https://docs.snowflake.com/en/user-guide/privatelink-azure)

There are some nuances for each connection, but in each case you will need a Snowflake administrator. As the Snowflake administrator, call the `SYSTEM$AUTHORIZE_STAGE_PRIVATELINK_ACCESS` function using the privateEndpointResourceID value as the function argument. This step authorizes access to the Snowflake internal stage through the Private Endpoint.

```sql

USE ROLE ACCOUNTADMIN;

-- AWS PrivateLink
SELECT SYSTEMS$AUTHORIZE_STATE_PRIVATELINK_ACCESS ( `AWS VPC ID` );

-- Azure Private Link
SELECT SYSTEMS$AUTHORIZE_STATE_PRIVATELINK_ACCESS ( `AZURE PRIVATED ENDPOINT RESOURCE ID` );

```

## Configuring Network Policies
If your organization uses [Snowflake Network Policies](https://docs.snowflake.com/en/user-guide/network-policies) to restrict access to your Snowflake account, you will need to add a network rule for dbt Cloud.

Expand Down Expand Up @@ -84,19 +108,23 @@ Open the Snowflake UI and take the following steps:
<Lightbox src="/img/docs/dbt-cloud/snowflakeprivatelink3.png" title="Update Network Policy"/>

### Using SQL

For quick and automated setup of network rules via SQL in Snowflake, the following commands allow you to create and configure access rules for dbt Cloud. These SQL examples demonstrate how to add a network rule and update your network policy accordingly.

1. Create a new network rule with the following SQL:
```sql

CREATE NETWORK RULE allow_dbt_cloud_access
MODE = INGRESS
TYPE = AWSVPCEID
VALUE_LIST = ('<VPCE_ID>'); -- Replace '<VPCE_ID>' with the actual ID provided

```

2. Add the rule to a network policy with the following SQL:
```sql

ALTER NETWORK POLICY <network_policy_name>
ADD ALLOWED_NETWORK_RULE_LIST =('allow_dbt_cloud_access');
```

```

0 comments on commit dd8c09c

Please sign in to comment.