Skip to content

Commit

Permalink
Update documentation on installing data apps (#950)
Browse files Browse the repository at this point in the history 
* Update documentation on installing data apps

* Improve documentation around states of data apps

Co-authored-by: Nick <[email protected]>

* update images

* add extra information about data model dependencies

* remove manual deploy section and replace with section about application connections

* update warehouse support info for data models

* Update install text

Co-authored-by: Nick <[email protected]>

* Add more information for data model running

Co-authored-by: Nick <[email protected]>

* Update creating connections

Co-authored-by: Nick <[email protected]>

* Update data model title

Co-authored-by: Nick <[email protected]>

* Update copy for installing applications

Co-authored-by: Nick <[email protected]>

* Update copy for apps

Co-authored-by: Nick <[email protected]>

* fix up image of install process

---------

Co-authored-by: Nick <[email protected]>
  • Loading branch information
@elliotstokes @stanch
elliotstokes and stanch authored Jul 15, 2024
1 parent dd1fdbc commit 320439b
Show file tree
Hide file tree
Showing 5 changed files with 35 additions and 160 deletions.
Binary file added docs/data-apps/images/add-connection.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/data-apps/images/install-app-tile.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/data-apps/images/sample-app-install.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
191 changes: 33 additions & 158 deletions docs/data-apps/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,44 @@ import DocCardList from '@theme/DocCardList';

## Accessing the Applications

You can find the Data Applications in the `Applications` tab in the left sidebar of your Snowplow Console. If the status is not `Live`, you can click on the tile and request access. A Snowplow Customer Success Manager will then get in contact with you to discuss the Snowplow Digital Analytics package.
You can find the Data Applications in the `Applications` tab in the left sidebar of your Snowplow Console. An application can be in one of three states:
* _By request_ — If you have not yet purchased a package that includes the application, you can click on “Learn more” and register your interest. A Snowplow Customer Success Manager will then get in contact with you to discuss getting access to the application.
* _Available_ — The application is ready to be set up. See the [installation instructions](#installing-applications) below.
* _Live_ — The application is ready to use.

Once the application is installed, clicking on the tile will launch the application in a separate browser tab. By default, anyone in your Console organization will be able to access data applications.

If you wish to invite others to use data applications but not have access to the rest of Console, you can [create a new user](/docs/using-the-snowplow-console/managing-users/adding-removing/index.md) and add a user with the `Data applications user` role. That user will then only see the Data Applications tab within Console. This permissions can be managed in the [usual way](/docs/using-the-snowplow-console/managing-users/managing-permissions/index.md).

## Installing applications

Once you have purchased an application, you are able to install it. Click the tile to begin installing the application and follow the steps.

![Image of application tile showing start install process](images/install-app-tile.png)

During installation, you will need to [provide a connection](#application-connections) to the warehouse you would like the application to use. It will also highlight any pipeline and data model dependencies that are required.

![Image of application install process](images/sample-app-install.png)

### Dependencies on data models

Generally, applications will depend on data models. If there are dependencies, then the installation flow will highlight which models are required and what models you currently have [running via BDP](/docs/modeling-your-data/running-data-models-via-snowplow-bdp/standard-models/index.md). It will also highlight any properties that you need to enable or configure for these data models.

:::note
If you are running the necessary data models yourself outside of BDP, then you will need to manually check that your setup satisfies the requirements for each application. These requirements are listed within the documentation pages for each application.
:::

## Application connections

To install a data application, you will need to create a connection to where your data model output or atomic table resides. You can do this as part of the installation process. Creating a connection involves securely setting up credentials for the data app to use to fetch the data.

:::tip
You might have already set up a warehouse connection for loading and/or modeling the data. Data applications, however, require a different connection, as you will often want them to run under a different user/role and with different permissions.
:::

![Image of application install process](images/add-connection.png)

Once you have selected a destination, provided the credentials and run the suggested SQL script, BDP will test the connection. Upon a successful test, the application will be available to use.

## General Usage
### Is the app running?
Expand All @@ -58,160 +90,3 @@ Our apps provide useful help text throughout the apps, keep an eye out for the h

### Log Out
If you wish to Log out of the application, you can do this from the sidebar. Note this also logs you out of Console.

## One off-setup
To setup the app and have it connect to your warehouse we require a user/role to run the app via, the below steps provide more information on this.

Once you have purchased the Snowplow Digital Analytics package, or other package containing Data Apps, Snowplow’s Infrastructure Support team will contact you to review the setup details, and then deploy the data applications into your cloud environment.

### Warehouse permissions

All data applications will require warehouse user credentials with read access to the tables powering the data applications. There are two options:

1. Reuse the Data Modeling User - in this case we will use the existing Data Modeling user you created during your on-boarding. Note that this user may need additional table access granted to it to use your own tables in any of the apps e.g. the [funnel builder](/docs/data-apps/funnel-builder/index.md).
2. [Recommended] Create a new Data Applications user - you can also create a new warehouse user with more fine-grained permissions. To create this, you can run the following script and then pass us the details via the secure messaging in console.

:::tip

While possible to re-use the same user for each data app, it may be beneficial to have a user per app for easier management of access and for logging purposes. In this case, please alter the scripts below to name each user/role as required.

:::


<Tabs groupId="warehouse" queryString>
<TabItem value="bigquery" label="BigQuery" default>


Create and share a GCP Service Account with us that has the following roles attached to it:

```hcl
data_app_roles = [
"roles/bigquery.dataViewer",
"roles/bigquery.filteredDataViewer",
"roles/bigquery.jobUser",
"roles/bigquery.metadataViewer",
"roles/bigquery.readSessionUser"
]
```

Once created share the following details back with us:

1. Project ID
2. BigQuery Dataset ID (indicator of where the app should default into, but is configurable in most apps once launched)
3. BigQuery Dataset Region (indicator of where the app should default into, but is configurable in most apps once launched)
4. BigQuery Service Account JSON (**please send this in a secure format**)

</TabItem>
<TabItem value="snowflake" label="Snowflake">


```sql
CREATE ROLE IF NOT EXISTS DATA_APPS_ROLE;

CREATE WAREHOUSE IF NOT EXISTS DATA_APPS_WH WITH
WAREHOUSE_SIZE = 'XSMALL' // NOTE: depends on your data volume
WAREHOUSE_TYPE = 'STANDARD'
AUTO_SUSPEND = 60
AUTO_RESUME = TRUE;
GRANT USAGE, OPERATE ON WAREHOUSE DATA_APPS_WH TO ROLE DATA_APPS_ROLE;

// NOTE: ensure you grant read-only access to any other database / schema
// you would like the applications to access.
GRANT USAGE ON DATABASE <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT USAGE ON FUTURE SCHEMAS IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT MONITOR ON FUTURE SCHEMAS IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT SELECT ON FUTURE TABLES IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT SELECT ON FUTURE VIEWS IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT USAGE ON ALL SCHEMAS IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT MONITOR ON ALL SCHEMAS IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT SELECT ON ALL TABLES IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;
GRANT SELECT ON ALL VIEWS IN database <database_with_snowplow_data> TO ROLE DATA_APPS_ROLE;

CREATE USER IF NOT EXISTS DATA_APPS_USER
PASSWORD='<user_password>'
MUST_CHANGE_PASSWORD = FALSE
DEFAULT_ROLE = DATA_APPS_ROLE
EMAIL = '<user_email>';
GRANT ROLE DATA_APPS_ROLE TO USER DATA_APPS_USER;
```

Once created share the following details back with us:

1. Snowflake Region
2. Snowflake Account (we need the account identifier which you can find following this guide: https://docs.snowflake.com/en/user-guide/admin-account-identifier#finding-the-organization-and-account-name-for-an-account)
3. Snowflake Username
4. Snowflake Password (**please send this in a secure format)**
5. Snowflake Database
6. Snowflake Schema (indicator of where the app should default into, but is configurable in most apps once launched)
7. Snowflake Warehouse
8. Snowflake Role

</TabItem>
<TabItem value="databricks" label="Databricks">

Create a new [Service Principal](https://docs.databricks.com/en/administration-guide/users-groups/service-principals.html#add-service-principals-to-your-account-using-the-account-console) in your workspace, make sure to give it Databricks SQL access and Workspace access in the configuration. Alternatively create a [new user](https://docs.databricks.com/en/administration-guide/users-groups/users.html#add-users-to-your-account-using-the-account-console) and ensure they have those accesses.

![Databricks account setup](./images/databricks-acc-setup.png)

```sql

// Uses the databricks inheritance model: https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/privileges.html#inheritance-model
// NOTE: ensure you grant read-only access to any other catalog / schema
// you would like the applications to access.
GRANT USE_CATALOG ON CATALOG <catalog_with_snowplow_data> TO <your_principal_or_user>;
GRANT USE_SCHEMA ON CATALOG <catalog_with_snowplow_data> TO <your_principal_or_user>;
GRANT SELECT ON CATALOG <catalog_with_snowplow_data> TO <your_principal_or_user>;
```

Once created share the following details back with us:

1. [Warehouse server hostname](https://docs.databricks.com/en/integrations/compute-details.html)
2. [The HTTP path to your SQL Warehouse](https://docs.databricks.com/en/integrations/compute-details.html)
3. Connection catalog (likely `hive_metastore`)
4. The default connection schema (indicator of where the app should default into, but is configurable in most apps once launched)
5. [One of](https://docs.databricks.com/en/administration-guide/users-groups/service-principals.html#manage-tokens-for-a-service-principal) (**please send this in a secure format**):
1. An OAUTH secret and client ID pair (Coming Soon)
2. A Personal access token

</TabItem>
<TabItem value="redshift" label="Redshift">


```sql
CREATE USER data_apps_user WITH PASSWORD '<data_apps_user_password>';
// NOTE: ensure you grant read-only access to any other schemas
// you would like the applications to access.
GRANT SELECT ON ALL TABLES IN SCHEMA <snowplow_schema> TO data_apps_user;
GRANT USAGE ON SCHEMA <snowplow_schema> TO data_apps_user;
// Ensure future grants by automatically granting access to created tables
ALTER DEFAULT PRIVILEGES FOR USER <user_creating_tables> IN SCHEMA <snowplow_schema> GRANT SELECT ON TABLES TO data_apps_user;
```

Once created share the following details back with us:

1. Redshift Username
2. Redshift Password (**please send this in a secure format**)
3. Redshift Database
4. Redshift Schema (indicator of where the app should default into, but is configurable in most apps once launched)
5. Redshift Host
6. Redshift Port (e.g. 5439)

If your Redshift is protected behind an SSH Tunnel please also share:

1. Bastion Host
2. Bastion Port (e.g. 22)
3. Bastion Username
4. SSH Private Key (**please send this in a secure format**)

</TabItem>

</Tabs>

:::tip

This grants read access to all tables within a schema for increased flexibility in applications but you can of course limit to just specific tables or views as required.

:::


Once you have passed us the user credentials, there may be some additional setup steps or requirements for each data app.
4 changes: 2 additions & 2 deletions ...odeling-your-data/running-data-models-via-snowplow-bdp/standard-models/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import TabItem from '@theme/TabItem';
[Standard data models](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/) are authored and maintained by Snowplow. Follow the steps below to create and run one.

## Create a warehouse connection
Begin by creating a new warehouse connection. It will be used by a data model to connect to your warehouse. Click on the "Set up new target" button on the [Data Models page](https://console.snowplowanalytics.com/data-modelling) and fill in all the necessary details.
Begin by creating a new warehouse connection. It will be used by a data model to connect to your warehouse. Click on the "Set up new connection" button on the [Data Models page](https://console.snowplowanalytics.com/data-models) and fill in all the necessary details.

![](images/warehouse-connections.png)

Expand All @@ -23,7 +23,7 @@ You might have already set up a warehouse connection for loading the data. Data

:::

Currently, only Snowflake and Databricks connections can be created. However, support for other warehouses will be added soon.
Supported destinations are Snowflake, Databricks, Redshift and BigQuery.

## Create a data model
To create a new data model, click the "Add data model" button. Then, select the data model you're interested in, such as [Unified Digital](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-models/dbt-unified-data-model/index.md), and the warehouse connection it should use. You can also specify the owners of the data model who will be alerted in case of failure.
Expand Down

0 comments on commit 320439b

Please sign in to comment.