From ed34bd67fa10109f68a383da8ba76ebfd96551e2 Mon Sep 17 00:00:00 2001 From: Jacob Cable <32874567+cabljac@users.noreply.github.com> Date: Mon, 16 Dec 2024 13:24:43 +0000 Subject: [PATCH] docs(firestore-bigquery-export): provided bash/ps1 scripts for granting cross-project access to the extension (#2241) --- firestore-bigquery-export/POSTINSTALL.md | 41 ++++++++++++-- firestore-bigquery-export/PREINSTALL.md | 39 ++++++++++++++ firestore-bigquery-export/README.md | 39 ++++++++++++++ .../scripts/grant-crossproject-access.ps1 | 43 +++++++++++++++ .../scripts/grant-crossproject-access.sh | 53 +++++++++++++++++++ 5 files changed, 210 insertions(+), 5 deletions(-) create mode 100644 firestore-bigquery-export/scripts/grant-crossproject-access.ps1 create mode 100755 firestore-bigquery-export/scripts/grant-crossproject-access.sh diff --git a/firestore-bigquery-export/POSTINSTALL.md b/firestore-bigquery-export/POSTINSTALL.md index 0e90782a3..96aa6278b 100644 --- a/firestore-bigquery-export/POSTINSTALL.md +++ b/firestore-bigquery-export/POSTINSTALL.md @@ -54,13 +54,44 @@ Enabling wildcard references will provide an additional STRING based column. The `Clustering` will not need to create or modify a table when adding clustering options, this will be updated automatically. -### Configuring Cross-Platform BigQuery Setup +#### Cross-project Streaming -When defining a specific BigQuery project ID, a manual step to set up permissions is required: +By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this: -1. Navigate to https://console.cloud.google.com/iam-admin/iam?project=${param:BIGQUERY_PROJECT_ID} -2. Add the **BigQuery Data Editor** role to the following service account: - `ext-${param:EXT_INSTANCE_ID}@${param:PROJECT_ID}.iam.gserviceaccount.com`. +1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID. + +2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts: + +**For Linux/Mac (Bash):** +```bash +curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh +chmod +x grant-crossproject-access.sh +./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID] +``` + +**For Windows (PowerShell):** +```powershell +Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1" +.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID] +``` + +**Parameters:** +For Bash script: +- `-f`: Your Firebase (source) project ID +- `-b`: Your target BigQuery project ID +- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export" + +For PowerShell script: +- `-FirebaseProject`: Your Firebase (source) project ID +- `-BigQueryProject`: Your target BigQuery project ID +- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export" + +**Prerequisites:** +- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured +- You must have permission to grant IAM roles on the target BigQuery project +- The extension must be installed before running the script + +**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts. ### _(Optional)_ Import existing documents diff --git a/firestore-bigquery-export/PREINSTALL.md b/firestore-bigquery-export/PREINSTALL.md index 88ca083cf..4bca364ab 100644 --- a/firestore-bigquery-export/PREINSTALL.md +++ b/firestore-bigquery-export/PREINSTALL.md @@ -100,6 +100,45 @@ If you follow these steps, your changelog table should be created using your cus After your data is in BigQuery, you can run the [schema-views script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md) (provided by this extension) to create views that make it easier to query relevant data. You only need to provide a JSON schema file that describes your data structure, and the schema-views script will create the views. +#### Cross-project Streaming + +By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this: + +1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID. + +2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts: + +**For Linux/Mac (Bash):** +```bash +curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh +chmod +x grant-crossproject-access.sh +./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID] +``` + +**For Windows (PowerShell):** +```powershell +Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1" +.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID] +``` + +**Parameters:** +For Bash script: +- `-f`: Your Firebase (source) project ID +- `-b`: Your target BigQuery project ID +- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export" + +For PowerShell script: +- `-FirebaseProject`: Your Firebase (source) project ID +- `-BigQueryProject`: Your target BigQuery project ID +- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export" + +**Prerequisites:** +- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured +- You must have permission to grant IAM roles on the target BigQuery project +- The extension must be installed before running the script + +**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts. + #### Billing To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing) diff --git a/firestore-bigquery-export/README.md b/firestore-bigquery-export/README.md index 81e28dbf3..b40ce5318 100644 --- a/firestore-bigquery-export/README.md +++ b/firestore-bigquery-export/README.md @@ -108,6 +108,45 @@ If you follow these steps, your changelog table should be created using your cus After your data is in BigQuery, you can run the [schema-views script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md) (provided by this extension) to create views that make it easier to query relevant data. You only need to provide a JSON schema file that describes your data structure, and the schema-views script will create the views. +#### Cross-project Streaming + +By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this: + +1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID. + +2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts: + +**For Linux/Mac (Bash):** +```bash +curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh +chmod +x grant-crossproject-access.sh +./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID] +``` + +**For Windows (PowerShell):** +```powershell +Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1" +.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID] +``` + +**Parameters:** +For Bash script: +- `-f`: Your Firebase (source) project ID +- `-b`: Your target BigQuery project ID +- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export" + +For PowerShell script: +- `-FirebaseProject`: Your Firebase (source) project ID +- `-BigQueryProject`: Your target BigQuery project ID +- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export" + +**Prerequisites:** +- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured +- You must have permission to grant IAM roles on the target BigQuery project +- The extension must be installed before running the script + +**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts. + #### Billing To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing) diff --git a/firestore-bigquery-export/scripts/grant-crossproject-access.ps1 b/firestore-bigquery-export/scripts/grant-crossproject-access.ps1 new file mode 100644 index 000000000..a2ed9b2a1 --- /dev/null +++ b/firestore-bigquery-export/scripts/grant-crossproject-access.ps1 @@ -0,0 +1,43 @@ +# Help message +function Show-Help { + Write-Host "Usage: .\grant-crossproject-access.ps1 -FirebaseProject -BigQueryProject [-ExtensionInstanceId ]" + Write-Host + Write-Host "Parameters:" + Write-Host " -FirebaseProject Firebase (source) project ID" + Write-Host " -BigQueryProject BigQuery project ID where dataset will be created" + Write-Host " -ExtensionInstanceId Extension instance ID (default: firestore-bigquery-export)" + exit 1 +} + +# Parameters +param( + [Parameter(Mandatory=$true)] + [string]$FirebaseProject, + + [Parameter(Mandatory=$true)] + [string]$BigQueryProject, + + [Parameter(Mandatory=$false)] + [string]$ExtensionInstanceId = "firestore-bigquery-export" +) + +# Construct service account email +$ServiceAccount = "ext-${ExtensionInstanceId}@${FirebaseProject}.iam.gserviceaccount.com" + +Write-Host "Using service account: $ServiceAccount" +Write-Host "Adding BigQuery permissions to $ServiceAccount on project: $BigQueryProject" + +$confirmation = Read-Host "Continue? (y/N)" +if ($confirmation -notmatch '^[yY]$') { + exit 1 +} + +# Grant bigquery.dataEditor role +gcloud projects add-iam-policy-binding $BigQueryProject ` + --member="serviceAccount:$ServiceAccount" ` + --role="roles/bigquery.dataEditor" + +# Grant bigquery.dataOwner which includes dataset.create permission +gcloud projects add-iam-policy-binding $BigQueryProject ` + --member="serviceAccount:$ServiceAccount" ` + --role="roles/bigquery.dataOwner" \ No newline at end of file diff --git a/firestore-bigquery-export/scripts/grant-crossproject-access.sh b/firestore-bigquery-export/scripts/grant-crossproject-access.sh new file mode 100755 index 000000000..57c4e5a27 --- /dev/null +++ b/firestore-bigquery-export/scripts/grant-crossproject-access.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +# Help message +function show_help { + echo "Usage: $0 -f FIREBASE_PROJECT -b BIGQUERY_PROJECT -i EXTENSION_INSTANCE_ID" + echo + echo "Options:" + echo " -f Firebase (source) project ID" + echo " -b BigQuery project ID where dataset will be created" + echo " -i Extension instance ID (default: firestore-bigquery-export)" + echo " -h Show this help message" + exit 1 +} + +# Set default extension instance ID +EXT_INSTANCE_ID="firestore-bigquery-export" + +# Parse command line arguments +while getopts "f:b:i:h" opt; do + case $opt in + f) FIREBASE_PROJECT="$OPTARG";; + b) BIGQUERY_PROJECT="$OPTARG";; + i) EXT_INSTANCE_ID="$OPTARG";; + h) show_help;; + ?) show_help;; + esac +done + +# Check if required arguments are provided +if [ -z "$FIREBASE_PROJECT" ] || [ -z "$BIGQUERY_PROJECT" ]; then + echo "Error: Both Firebase and BigQuery project IDs are required" + show_help +fi + +# Construct service account email +SERVICE_ACCOUNT="ext-${EXT_INSTANCE_ID}@${FIREBASE_PROJECT}.iam.gserviceaccount.com" + +echo "Using service account: $SERVICE_ACCOUNT" +echo "Adding BigQuery permissions to $SERVICE_ACCOUNT on project: $BIGQUERY_PROJECT" +read -p "Continue? (y/N) " -n 1 -r +if [[ ! $REPLY =~ ^[Yy]$ ]]; then + exit 1 +fi + +# Grant bigquery.dataEditor role +gcloud projects add-iam-policy-binding $BIGQUERY_PROJECT \ + --member="serviceAccount:$SERVICE_ACCOUNT" \ + --role="roles/bigquery.dataEditor" + +# Grant bigquery.dataOwner which includes dataset.create permission +gcloud projects add-iam-policy-binding $BIGQUERY_PROJECT \ + --member="serviceAccount:$SERVICE_ACCOUNT" \ + --role="roles/bigquery.dataOwner" \ No newline at end of file