Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added OpenShift GCP supplemental values #237

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Added OpenShift GCP supplemental values #237

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
062274b
add gcp supplemental values
Dec 11, 2024
518e575
add gcp supplemental values
Dec 11, 2024
c9c151e
fix: split the docs into gcp and os
Dec 12, 2024
ddbe56b
add comments
Dec 13, 2024
fe4c31f
resolve comments
Dec 16, 2024
c722457
add comments
Dec 17, 2024
0b9df67
fix comments and remove dfs
Dec 17, 2024
228753e
fix comments
Dec 18, 2024
bdb62ad
change database config for grafana
Dec 18, 2024
4c0d10b
fix: grafana database
Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
385 changes: 385 additions & 0 deletions getting-started/templates/GCP/gcp-supplemental-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,385 @@
global:
ingress:
api:
annotations:
kubernetes.io/ingress.class: <user-defined-ingress-class> # <ATTENTION> - Enter the ingress class used
ui:
annotations:
kubernetes.io/ingress.class: <user-defined-ingress-class> # <ATTENTION> - Enter the ingress class used

testmonitorservice:
## Cloud SQL auth proxy sidecar container to authenticate to Cloud SQL Postgres database
## ref: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#run_the_in_a_sidecar_pattern
sidecars:
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
- name: cloud-sql-auth-proxy
image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.8.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets remove the tag from here and replace with a placeholder and attention comment

volumeMounts:
# This volume mount is required for the proxy to authenticate with cloudSQL using a service account key file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this token volume mount only required for workload identity?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, added comment for the same

- name: <token-volume-name> # <ATTENTION> - Enter the volume name where the token is mounted
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
mountPath: <token-mount-path> # <ATTENTION> - Enter the path where the token is mounted
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
readOnly: true
# This volume mount is required for the proxy to authenticate with cloudSQL when using Workload Identity Federation.
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
- name: <config-volume-name> # <ATTENTION> - Enter the volume name where config.json is mounted
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
mountPath: /secrets/
readOnly: true
env:
- name: "GOOGLE_APPLICATION_CREDENTIALS"
value: /secrets/<secret-key> # <ATTENTION> - - Enter the key which was used while creating the secret
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
args:
# If connecting from a VPC-native GKE cluster, you can use the
# following flag to have the proxy connect over private IP
- "--private-ip"

# If you are not connecting with Automatic IAM, you can delete the following flag.
- "--auto-iam-authn"

# Enable structured logging with LogEntry format:
- "--structured-logs"

# Ensures the proxy exits gracefully with a 0 exit code when it receives a SIGTERM signal
- "--exit-zero-on-sigterm"

- "--port=5432"
- "<connection-name>" # <ATTENTION> - Enter the connection name from the CloudSQL instance

# The credentials file is required for the proxy to authenticate using a service account key file.
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
- "--credentials-file=/secrets/<secret-key>" # <ATTENTION> - Enter the key which was used while creating the secret
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
securityContext:
# The default Cloud SQL Auth Proxy image runs as the
# "nonroot" user and group (uid: 65532) by default.
runAsNonRoot: true
restartPolicy: Always
# You should use resource requests/limits as a best practice to prevent
# pods from consuming too many resources and affecting the execution of
# other pods. You should adjust the following values based on what your
# application needs. For details, see
# https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
resources:
requests:
# The proxy's memory use scales linearly with the number of active
# connections. Fewer open connections will use less memory. Adjust
# this value based on your application's requirements.
memory: "1Gi"
# The proxy's CPU use scales linearly with the amount of IO between
# the database and the application. Adjust this value based on your
# application's requirements.
cpu: "1"

## Extra volumes that can be used in sidecars
extraVolumes:
# This volume is required for the proxy to authenticate with cloudSQL using a service account key file.
- name: <token-volume-name> # <ATTENTION> - Enter the volume name where the token is to be mounted
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
projected:
sources:
- serviceAccountToken:
audience: <audience-name> # <ATTENTION> - Enter the audience name for the projected service account token
expirationSeconds: 3600
path: token
# This volume is required for the proxy to authenticate with cloudSQL when using Workload Identity Federation.
- name: <config-volume-name> # <ATTENTION> - Enter the volume name where config.json is to be mounted
secret:
secretName: <secret-name> # <ATTENTION> - Enter the secret name where config.json is added.

connectionInfo:
## @param database.connectionInfo.host PostgreSQL hostname. Since the cloud-sql-auth-proxy is used, this value should be localhost.
##
host: "localhost"
## @param database.connectionInfo.port PostgreSQL port
##
port: "5432"
## @param database.connectionInfo.dbName PostgreSQL database name
##
dbName: <database-name> # <ATTENTION> - Enter the PostgreSQL database name for testmonitor service
## @param database.connectionInfo.user PostgreSQL username used by the service
##
user: <username> # <ATTENTION> - Enter the PostgreSQL username for the testmonitor service
## @param database.connectionInfo.secretName The name of an existing secret with PostgreSQL connection credentials
##
secretName: <secret-name> # <ATTENTION> - Enter the secret name for the testmonitor service with PostgreSQL connection credentials
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
## @param database.connectionInfo.passwordKey Password key for database.connectionInfo.user to be retrieved from existing secret
## NOTE: Ignored unless `database.connectionInfo.secretName` parameter is set.
##
# passwordKey: "passkey"
## @param database.connectionInfo.migrationPasswordKey Password key for database.connectionInfo.migrationUser to be retrieved from existing secret
## If unset database.connectionInfo.passwordKey is used instead.
## NOTE: Ignored unless `database.connectionInfo.secretName` parameter is set.
##
# migrationPasswordKey: null

serviceAccount:
## @param serviceAccount.annotations Annotations to add to the service account
##
annotations:
{
iam.gke.io/gcp-service-account=<YOUR-GSA-NAME>@<YOUR-GOOGLE-CLOUD-PROJECT>.iam.gserviceaccount.com, # <ATTENTION> - Enter the CloudSQL service account name and the Google Cloud Project name
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
}
## @param serviceAccount.name The name of the service account to use.
## If not set and create is true, a name is generated using the fullname template
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
##
name: ""

dashboardhost:
grafana:
## Configuring Cloud SQL auth proxy extra container to authenticate to Cloud SQL Postgres database
## ref: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#run_the_in_a_sidecar_pattern
extraContainers: |
- name: cloud-sql-auth-proxy
image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.8.0
volumeMounts:
# This volume mount is required for the proxy to authenticate with cloudSQL using a service account key file.
- name: <token-volume-name> # <ATTENTION> - Enter the volume name where the token is mounted
mountPath: <token-mount-path> # <ATTENTION> - Enter the path where the token is mounted
readOnly: true
# This volume mount is required for the proxy to authenticate with cloudSQL when using Workload Identity Federation.
- name: <config-volume-name> # <ATTENTION> - Enter the volume name where config.json is mounted
mountPath: /secrets/
readOnly: true
env:
- name: "GOOGLE_APPLICATION_CREDENTIALS"
value: /secrets/<secret-key> # <ATTENTION> - Enter the key which was used while creating the secret
args:
# If connecting from a VPC-native GKE cluster, you can use the
# following flag to have the proxy connect over private IP
- "--private-ip"

# If you are not connecting with Automatic IAM, you can delete the following flag.
- "--auto-iam-authn"

# Enable structured logging with LogEntry format:
- "--structured-logs"

# Ensures the proxy exits gracefully with a 0 exit code when it receives a SIGTERM signal
- "--exit-zero-on-sigterm"

- "--port=5432"
- "<connection-name>" # <ATTENTION> - Enter the connection name from the CloudSQL instance

# The credentials file is required for the proxy to authenticate using a service account key file.
- "--credentials-file=/secrets/<secret-key>" # <ATTENTION> - Enter the key which was used while creating the secret
securityContext:
# The default Cloud SQL Auth Proxy image runs as the
# "nonroot" user and group (uid: 65532) by default.
runAsNonRoot: true
restartPolicy: Always
# You should use resource requests/limits as a best practice to prevent
# pods from consuming too many resources and affecting the execution of
# other pods. You should adjust the following values based on what your
# application needs. For details, see
# https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
resources:
requests:
# The proxy's memory use scales linearly with the number of active
# connections. Fewer open connections will use less memory. Adjust
# this value based on your application's requirements.
memory: "1Gi"
# The proxy's CPU use scales linearly with the amount of IO between
# the database and the application. Adjust this value based on your
# application's requirements.
cpu: "1"

## Extra volumes that can be used in extrContainers
extraContainerVolumes:
# This volume is required for the proxy to authenticate with cloudSQL using a service account key file.
- name: <token-volume-name> # <ATTENTION> - Enter the volume name where the token is to be mounted
projected:
sources:
- serviceAccountToken:
audience: <audience-name> # <ATTENTION> - Enter the audience name for the projected service account token
expirationSeconds: 3600
path: token
# This volume is required for the proxy to authenticate with cloudSQL when using Workload Identity Federation.
- name: <config-volume-name> # <ATTENTION> - Enter the volume name where config.json is to be mounted
secret:
secretName: <secret-name> # <ATTENTION> - Enter the secret name where config.json is added.

connectionInfo:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like these are not the right values for grafana.

Check here

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The values for grafana database have been modified

## @param database.connectionInfo.host PostgreSQL hostname. Since the cloud-sql-auth-proxy is used, this value should be localhost.
##
host: "localhost"
## @param database.connectionInfo.port PostgreSQL port
##
port: "5432"
## @param database.connectionInfo.dbName PostgreSQL database name
##
dbName: <database-name> # <ATTENTION> - Enter the PostgreSQL database name for testmonitor service
## @param database.connectionInfo.user PostgreSQL username used by the service
##
user: <username> # <ATTENTION> - Enter the PostgreSQL username for the testmonitor service
## @param database.connectionInfo.secretName The name of an existing secret with
## PostgreSQL connection credentials
##
secretName: <secret-name> # <ATTENTION> - Enter the secret name for the testmonitor service with PostgreSQL connection credentials
## @param database.connectionInfo.passwordKey Password key for database.connectionInfo.user to be retrieved from existing secret
## NOTE: Ignored unless `database.connectionInfo.secretName` parameter is set.
##
# passwordKey: "passkey"
## @param database.connectionInfo.migrationPasswordKey Password key for database.connectionInfo.migrationUser to be retrieved from existing secret
## If unset database.connectionInfo.passwordKey is used instead.
## NOTE: Ignored unless `database.connectionInfo.secretName` parameter is set.
##
# migrationPasswordKey: null

serviceAccount:
## @param serviceAccount.annotations Annotations to add to the service account
##
annotations:
{
iam.gke.io/gcp-service-account=<YOUR-GSA-NAME>@<YOUR-GOOGLE-CLOUD-PROJECT>.iam.gserviceaccount.com, # <ATTENTION> - Enter the CloudSQL service account name and the Google Cloud Project name
}
## @param serviceAccount.name The name of the service account to use.
## If not set and create is true, a name is generated using the fullname template
##
name: ""

dynamicformfields:
## Configuring Cloud SQL auth proxy sidecar container to authenticate to Cloud SQL Postgres database
## ref: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine#run_the_in_a_sidecar_pattern
sidecars:
- name: cloud-sql-auth-proxy
image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.8.0
volumeMounts:
# This volume mount is required for the proxy to authenticate with cloudSQL using a service account key file.
- name: <token-volume-name> # <ATTENTION> - Enter the volume name where the token is mounted
mountPath: <token-mount-path> # <ATTENTION> - Enter the path where the token is mounted
readOnly: true
# This volume mount is required for the proxy to authenticate with cloudSQL when using Workload Identity Federation.
- name: <config-volume-name> # <ATTENTION> - Enter the volume name where config.json is mounted
mountPath: /secrets/
readOnly: true
env:
- name: "GOOGLE_APPLICATION_CREDENTIALS"
value: /secrets/<secret-key> # <ATTENTION> - Enter the key which was used while creating the secret
args:
# If connecting from a VPC-native GKE cluster, you can use the
# following flag to have the proxy connect over private IP
- "--private-ip"

# If you are not connecting with Automatic IAM, you can delete the following flag.
- "--auto-iam-authn"

# Enable structured logging with LogEntry format:
- "--structured-logs"

# Ensures the proxy exits gracefully with a 0 exit code when it receives a SIGTERM signal
- "--exit-zero-on-sigterm"

- "--port=5432"
- "<connection-name>" # <ATTENTION> - Enter the connection name from the CloudSQL instance

# The credentials file is required for the proxy to authenticate using a service account key file.
- "--credentials-file=/secrets/<secret-key>" # <ATTENTION> - Enter the key which was used while creating the secret
securityContext:
# The default Cloud SQL Auth Proxy image runs as the
# "nonroot" user and group (uid: 65532) by default.
runAsNonRoot: true
restartPolicy: Always
# You should use resource requests/limits as a best practice to prevent
# pods from consuming too many resources and affecting the execution of
# other pods. You should adjust the following values based on what your
# application needs. For details, see
# https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
resources:
requests:
# The proxy's memory use scales linearly with the number of active
# connections. Fewer open connections will use less memory. Adjust
# this value based on your application's requirements.
memory: "1Gi"
# The proxy's CPU use scales linearly with the amount of IO between
# the database and the application. Adjust this value based on your
# application's requirements.
cpu: "1"

## Extra volumes that can be used in sidecars
extraVolumes:
# This volume is required for the proxy to authenticate with cloudSQL using a service account key file.
- name: <token-volume-name> # <ATTENTION> - Enter the volume name where the token is to be mounted
projected:
sources:
- serviceAccountToken:
audience: <audience-name> # <ATTENTION> - Enter the audience name for the projected service account token
expirationSeconds: 3600
path: token
# This volume is required for the proxy to authenticate with cloudSQL when using Workload Identity Federation.
- name: <config-volume-name> # <ATTENTION> - Enter the volume name where config.json is to be mounted
secret:
secretName: <secret-name> # <ATTENTION> - Enter the secret name where config.json is added.

connectionInfo:
## @param database.connectionInfo.host PostgreSQL hostname. Since the cloud-sql-auth-proxy is used, this value should be localhost.
##
host: "localhost"
## @param database.connectionInfo.port PostgreSQL port
##
port: "5432"
## @param database.connectionInfo.dbName PostgreSQL database name
##
dbName: <database-name> # <ATTENTION> - Enter the PostgreSQL database name for testmonitor service
## @param database.connectionInfo.user PostgreSQL username used by the service
##
user: <username> # <ATTENTION> - Enter the PostgreSQL username for the testmonitor service
## @param database.connectionInfo.secretName The name of an existing secret with
## PostgreSQL connection credentials
##
secretName: <secret-name> # <ATTENTION> - Enter the secret name for the testmonitor service with PostgreSQL connection credentials
## @param database.connectionInfo.passwordKey Password key for database.connectionInfo.user to be retrieved from existing secret
## NOTE: Ignored unless `database.connectionInfo.secretName` parameter is set.
##
# passwordKey: "passkey"
## @param database.connectionInfo.migrationPasswordKey Password key for database.connectionInfo.migrationUser to be retrieved from existing secret
## If unset database.connectionInfo.passwordKey is used instead.
## NOTE: Ignored unless `database.connectionInfo.secretName` parameter is set.
##
# migrationPasswordKey: null

serviceAccount:
## @param serviceAccount.annotations Annotations to add to the service account
##
annotations:
{
iam.gke.io/gcp-service-account=<YOUR-GSA-NAME>@<YOUR-GOOGLE-CLOUD-PROJECT>.iam.gserviceaccount.com, # <ATTENTION> - Enter the CloudSQL service account name and the Google Cloud Project name
}
## @param serviceAccount.name The name of the service account to use.
## If not set and create is true, a name is generated using the fullname template
##
name: ""

dataframeservice:
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
s3:
port: 443
bucket: <name-of-gcs-bucket> # <ATTENTION> - Enter the name of the GCS bucket for dataframe service
scheme: "https://"
host: "storage.googleapis.com"
region: "us-east1"

fileingestion:
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved
s3:
port: 443
bucket: <name-of-gcs-bucket> # <ATTENTION> - Enter the name of the GCS bucket for fileingestion service
scheme: "https://"
host: "storage.googleapis.com"
region: "us-east1"
priyadarshini-ni marked this conversation as resolved.
Show resolved Hide resolved

feedservice:
s3:
port: 443
bucket: <name-of-gcs-bucket> # <ATTENTION> - Enter the name of the GCS bucket for feedservice service
scheme: "https://"
host: "storage.googleapis.com"
region: "us-east1"

saltmaster:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this expected to change based on the cloud provider?

serviceTCP:
type: LoadBalancer

nbexecservice:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nbexec (has direct s3 dependency), dfs and dremio in dfs too have GCS dependency. Can we also include them?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added s3 dependency for nbexecservice. The GCS guide has not been included in this PR for DFS as the support is not fully available yet. Refer PR description for more info.

argo:
## Configure GCS access.
##
artifactRepository:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s3 dependency for Argo in nbexec is no longer required. We can remove this

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

s3:
port: 443
bucket: <name-of-gcs-bucket> # <ATTENTION> - Enter the name of the GCS bucket for nbexecservice service
scheme: "https://"
host: "storage.googleapis.com"
region: "us-east1"
insecure: false
Loading
Loading