Skip to content

Commit

Permalink
docs(firestore-bigquery-export): provided bash/ps1 scripts for granti…
Browse files Browse the repository at this point in the history
…ng cross-project access to the extension (#2241)
  • Loading branch information
cabljac authored Dec 16, 2024
1 parent f87b847 commit ed34bd6
Show file tree
Hide file tree
Showing 5 changed files with 210 additions and 5 deletions.
41 changes: 36 additions & 5 deletions firestore-bigquery-export/POSTINSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,13 +54,44 @@ Enabling wildcard references will provide an additional STRING based column. The
`Clustering` will not need to create or modify a table when adding clustering options, this will be updated automatically.
### Configuring Cross-Platform BigQuery Setup
#### Cross-project Streaming
When defining a specific BigQuery project ID, a manual step to set up permissions is required:
By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this:
1. Navigate to https://console.cloud.google.com/iam-admin/iam?project=${param:BIGQUERY_PROJECT_ID}
2. Add the **BigQuery Data Editor** role to the following service account:
`ext-${param:EXT_INSTANCE_ID}@${param:PROJECT_ID}.iam.gserviceaccount.com`.
1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID.
2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts:
**For Linux/Mac (Bash):**
```bash
curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh
chmod +x grant-crossproject-access.sh
./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID]
```

**For Windows (PowerShell):**
```powershell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1"
.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID]
```

**Parameters:**
For Bash script:
- `-f`: Your Firebase (source) project ID
- `-b`: Your target BigQuery project ID
- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"

For PowerShell script:
- `-FirebaseProject`: Your Firebase (source) project ID
- `-BigQueryProject`: Your target BigQuery project ID
- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"

**Prerequisites:**
- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured
- You must have permission to grant IAM roles on the target BigQuery project
- The extension must be installed before running the script

**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.

### _(Optional)_ Import existing documents

Expand Down
39 changes: 39 additions & 0 deletions firestore-bigquery-export/PREINSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,45 @@ If you follow these steps, your changelog table should be created using your cus

After your data is in BigQuery, you can run the [schema-views script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md) (provided by this extension) to create views that make it easier to query relevant data. You only need to provide a JSON schema file that describes your data structure, and the schema-views script will create the views.

#### Cross-project Streaming

By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this:

1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID.

2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts:

**For Linux/Mac (Bash):**
```bash
curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh
chmod +x grant-crossproject-access.sh
./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID]
```

**For Windows (PowerShell):**
```powershell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1"
.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID]
```

**Parameters:**
For Bash script:
- `-f`: Your Firebase (source) project ID
- `-b`: Your target BigQuery project ID
- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"

For PowerShell script:
- `-FirebaseProject`: Your Firebase (source) project ID
- `-BigQueryProject`: Your target BigQuery project ID
- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"

**Prerequisites:**
- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured
- You must have permission to grant IAM roles on the target BigQuery project
- The extension must be installed before running the script

**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.

#### Billing
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)

Expand Down
39 changes: 39 additions & 0 deletions firestore-bigquery-export/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,45 @@ If you follow these steps, your changelog table should be created using your cus

After your data is in BigQuery, you can run the [schema-views script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md) (provided by this extension) to create views that make it easier to query relevant data. You only need to provide a JSON schema file that describes your data structure, and the schema-views script will create the views.

#### Cross-project Streaming

By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this:

1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID.

2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts:

**For Linux/Mac (Bash):**
```bash
curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh
chmod +x grant-crossproject-access.sh
./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID]
```

**For Windows (PowerShell):**
```powershell
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1"
.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID]
```

**Parameters:**
For Bash script:
- `-f`: Your Firebase (source) project ID
- `-b`: Your target BigQuery project ID
- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"

For PowerShell script:
- `-FirebaseProject`: Your Firebase (source) project ID
- `-BigQueryProject`: Your target BigQuery project ID
- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"

**Prerequisites:**
- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured
- You must have permission to grant IAM roles on the target BigQuery project
- The extension must be installed before running the script

**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.

#### Billing
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)

Expand Down
43 changes: 43 additions & 0 deletions firestore-bigquery-export/scripts/grant-crossproject-access.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Help message
function Show-Help {
Write-Host "Usage: .\grant-crossproject-access.ps1 -FirebaseProject <PROJECT_ID> -BigQueryProject <PROJECT_ID> [-ExtensionInstanceId <INSTANCE_ID>]"
Write-Host
Write-Host "Parameters:"
Write-Host " -FirebaseProject Firebase (source) project ID"
Write-Host " -BigQueryProject BigQuery project ID where dataset will be created"
Write-Host " -ExtensionInstanceId Extension instance ID (default: firestore-bigquery-export)"
exit 1
}

# Parameters
param(
[Parameter(Mandatory=$true)]
[string]$FirebaseProject,

[Parameter(Mandatory=$true)]
[string]$BigQueryProject,

[Parameter(Mandatory=$false)]
[string]$ExtensionInstanceId = "firestore-bigquery-export"
)

# Construct service account email
$ServiceAccount = "ext-${ExtensionInstanceId}@${FirebaseProject}.iam.gserviceaccount.com"

Write-Host "Using service account: $ServiceAccount"
Write-Host "Adding BigQuery permissions to $ServiceAccount on project: $BigQueryProject"

$confirmation = Read-Host "Continue? (y/N)"
if ($confirmation -notmatch '^[yY]$') {
exit 1
}

# Grant bigquery.dataEditor role
gcloud projects add-iam-policy-binding $BigQueryProject `
--member="serviceAccount:$ServiceAccount" `
--role="roles/bigquery.dataEditor"

# Grant bigquery.dataOwner which includes dataset.create permission
gcloud projects add-iam-policy-binding $BigQueryProject `
--member="serviceAccount:$ServiceAccount" `
--role="roles/bigquery.dataOwner"
53 changes: 53 additions & 0 deletions firestore-bigquery-export/scripts/grant-crossproject-access.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/bin/bash

# Help message
function show_help {
echo "Usage: $0 -f FIREBASE_PROJECT -b BIGQUERY_PROJECT -i EXTENSION_INSTANCE_ID"
echo
echo "Options:"
echo " -f Firebase (source) project ID"
echo " -b BigQuery project ID where dataset will be created"
echo " -i Extension instance ID (default: firestore-bigquery-export)"
echo " -h Show this help message"
exit 1
}

# Set default extension instance ID
EXT_INSTANCE_ID="firestore-bigquery-export"

# Parse command line arguments
while getopts "f:b:i:h" opt; do
case $opt in
f) FIREBASE_PROJECT="$OPTARG";;
b) BIGQUERY_PROJECT="$OPTARG";;
i) EXT_INSTANCE_ID="$OPTARG";;
h) show_help;;
?) show_help;;
esac
done

# Check if required arguments are provided
if [ -z "$FIREBASE_PROJECT" ] || [ -z "$BIGQUERY_PROJECT" ]; then
echo "Error: Both Firebase and BigQuery project IDs are required"
show_help
fi

# Construct service account email
SERVICE_ACCOUNT="ext-${EXT_INSTANCE_ID}@${FIREBASE_PROJECT}.iam.gserviceaccount.com"

echo "Using service account: $SERVICE_ACCOUNT"
echo "Adding BigQuery permissions to $SERVICE_ACCOUNT on project: $BIGQUERY_PROJECT"
read -p "Continue? (y/N) " -n 1 -r
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
exit 1
fi

# Grant bigquery.dataEditor role
gcloud projects add-iam-policy-binding $BIGQUERY_PROJECT \
--member="serviceAccount:$SERVICE_ACCOUNT" \
--role="roles/bigquery.dataEditor"

# Grant bigquery.dataOwner which includes dataset.create permission
gcloud projects add-iam-policy-binding $BIGQUERY_PROJECT \
--member="serviceAccount:$SERVICE_ACCOUNT" \
--role="roles/bigquery.dataOwner"

0 comments on commit ed34bd6

Please sign in to comment.