Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slurm on GKE - Guide #864

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Update providers.tf

da30a6b
Select commit
Loading
Failed to load commit list.
Open

Slurm on GKE - Guide #864

Update providers.tf
da30a6b
Select commit
Loading
Failed to load commit list.
Google Cloud Build / pr-review-trigger-ap-clusters (gke-ai-eco-dev) failed Nov 16, 2024 in 33m 55s

Summary

Build Information

Trigger pr-review-trigger-ap-clusters
Build 7e3e68a4-7e05-4066-9c33-44d18571eb42
Start 2024-11-15T15:53:35-08:00
Duration 33m16.429s
Status FAILURE

Steps

Step Status Duration
validate platform SUCCESS 50.075s
validate ray SUCCESS 16.303s
validate jupyterhub SUCCESS 16.306s
validate rag SUCCESS 16.86s
create gke cluster SUCCESS 10m17.654s
test ray cluster SUCCESS 4m42.594s
cleanup ray cluster SUCCESS 33.478s
test jupyterhub SUCCESS 3m8.599s
cleanup jupyterhub SUCCESS 46.714s
test rag FAILURE 14m47.635s
cleanup rag SUCCESS 2m5.099s
cleanup gke cluster FAILURE 4m53.946s
check result FAILURE 612ms

Details

starting build "7e3e68a4-7e05-4066-9c33-44d18571eb42"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/GoogleCloudPlatform/ai-on-gke
 * branch            da30a6b33263053cbcc0a8a0230b93163b570c43 -> FETCH_HEAD
HEAD is now at da30a6b Update providers.tf
SETUPBUILD
BUILD
Starting Step #0 - "validate platform"
Step #0 - "validate platform": Pulling image: gcr.io/gke-ai-eco-dev/terraform
Step #0 - "validate platform": Using default tag: latest
Step #0 - "validate platform": latest: Pulling from gke-ai-eco-dev/terraform
Step #0 - "validate platform": ddf874abf16c: Pulling fs layer
Step #0 - "validate platform": ecfb911cf53f: Pulling fs layer
Step #0 - "validate platform": ce689bb70bd5: Pulling fs layer
Step #0 - "validate platform": 7d2b54998739: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Pulling fs layer
Step #0 - "validate platform": cc032834b94b: Pulling fs layer
Step #0 - "validate platform": 0279610fca41: Pulling fs layer
Step #0 - "validate platform": 6f125b563dda: Pulling fs layer
Step #0 - "validate platform": 7c8b6e22fcff: Pulling fs layer
Step #0 - "validate platform": faaaa1840355: Pulling fs layer
Step #0 - "validate platform": 60a28c2199b1: Pulling fs layer
Step #0 - "validate platform": 4c7171ac56db: Pulling fs layer
Step #0 - "validate platform": 372d85ec2f1f: Pulling fs layer
Step #0 - "validate platform": 633b411e5bd5: Pulling fs layer
Step #0 - "validate platform": 7d2b54998739: Waiting
Step #0 - "validate platform": 6f125b563dda: Waiting
Step #0 - "validate platform": faaaa1840355: Waiting
Step #0 - "validate platform": b1c39b09da0f: Waiting
Step #0 - "validate platform": 60a28c2199b1: Waiting
Step #0 - "validate platform": 7c8b6e22fcff: Waiting
Step #0 - "validate platform": cc032834b94b: Waiting
Step #0 - "validate platform": 4c7171ac56db: Waiting
Step #0 - "validate platform": 0279610fca41: Waiting
Step #0 - "validate platform": 633b411e5bd5: Waiting
Step #0 - "validate platform": ecfb911cf53f: Verifying Checksum
Step #0 - "validate platform": ecfb911cf53f: Download complete
Step #0 - "validate platform": ce689bb70bd5: Verifying Checksum
Step #0 - "validate platform": ce689bb70bd5: Download complete
Step #0 - "validate platform": 7d2b54998739: Download complete
Step #0 - "validate platform": ddf874abf16c: Verifying Checksum
Step #0 - "validate platform": ddf874abf16c: Download complete
Step #0 - "validate platform": 0279610fca41: Verifying Checksum
Step #0 - "validate platform": 0279610fca41: Download complete
Step #0 - "validate platform": 6f125b563dda: Verifying Checksum
Step #0 - "validate platform": 6f125b563dda: Download complete
Step #0 - "validate platform": 7c8b6e22fcff: Verifying Checksum
Step #0 - "validate platform": 7c8b6e22fcff: Download complete
Step #0 - "validate platform": cc032834b94b: Verifying Checksum
Step #0 - "validate platform": cc032834b94b: Download complete
Step #0 - "validate platform": faaaa1840355: Verifying Checksum
Step #0 - "validate platform": faaaa1840355: Download complete
Step #0 - "validate platform": 4c7171ac56db: Verifying Checksum
Step #0 - "validate platform": 4c7171ac56db: Download complete
Step #0 - "validate platform": 372d85ec2f1f: Download complete
Step #0 - "validate platform": ddf874abf16c: Pull complete
Step #0 - "validate platform": 60a28c2199b1: Verifying Checksum
Step #0 - "validate platform": 60a28c2199b1: Download complete
Step #0 - "validate platform": ecfb911cf53f: Pull complete
Step #0 - "validate platform": ce689bb70bd5: Pull complete
Step #0 - "validate platform": 7d2b54998739: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Verifying Checksum
Step #0 - "validate platform": 633b411e5bd5: Download complete
Step #0 - "validate platform": b1c39b09da0f: Verifying Checksum
Step #0 - "validate platform": b1c39b09da0f: Download complete
Step #0 - "validate platform": b1c39b09da0f: Pull complete
Step #0 - "validate platform": cc032834b94b: Pull complete
Step #0 - "validate platform": 0279610fca41: Pull complete
Step #0 - "validate platform": 6f125b563dda: Pull complete
Step #0 - "validate platform": 7c8b6e22fcff: Pull complete
Step #0 - "validate platform": faaaa1840355: Pull complete
Step #0 - "validate platform": 60a28c2199b1: Pull complete
Step #0 - "validate platform": 4c7171ac56db: Pull complete
Step #0 - "validate platform": 372d85ec2f1f: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Pull complete
Step #0 - "validate platform": Digest: sha256:202db292b551d530ff51210759dd25971cb64b5afe675a5616e0c244d48a452b
Step #0 - "validate platform": Status: Downloaded newer image for gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing the backend...
Step #0 - "validate platform": Initializing modules...
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for cloud-nat...
Step #0 - "validate platform": - cloud-nat in .terraform/modules/cloud-nat
Step #0 - "validate platform": - custom-network in ../modules/gcp-network
Step #0 - "validate platform": - private-gke-autopilot-cluster in ../modules/gke-autopilot-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for private-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - private-gke-autopilot-cluster.gke in .terraform/modules/private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #0 - "validate platform": - private-gke-standard-cluster in ../modules/gke-standard-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for private-gke-standard-cluster.gke...
Step #0 - "validate platform": - private-gke-standard-cluster.gke in .terraform/modules/private-gke-standard-cluster.gke/modules/private-cluster
Step #0 - "validate platform": - public-gke-autopilot-cluster in ../modules/gke-autopilot-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for public-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - public-gke-autopilot-cluster.gke in .terraform/modules/public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #0 - "validate platform": - public-gke-standard-cluster in ../modules/gke-standard-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for public-gke-standard-cluster.gke...
Step #0 - "validate platform": - public-gke-standard-cluster.gke in .terraform/modules/public-gke-standard-cluster.gke
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing provider plugins...
Step #0 - "validate platform": - Finding hashicorp/kubernetes versions matching "~> 2.10, 2.18.1"...
Step #0 - "validate platform": - Finding hashicorp/google versions matching ">= 4.51.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #0 - "validate platform": - Finding hashicorp/google-beta versions matching ">= 5.40.0, != 5.44.0, <= 6.1.0, < 7.0.0"...
Step #0 - "validate platform": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #0 - "validate platform": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #0 - "validate platform": - Installing hashicorp/helm v2.8.0...
Step #0 - "validate platform": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/random v3.6.3...
Step #0 - "validate platform": - Installed hashicorp/random v3.6.3 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/kubernetes v2.18.1...
Step #0 - "validate platform": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google v5.45.0...
Step #0 - "validate platform": - Installed hashicorp/google v5.45.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google-beta v6.1.0...
Step #0 - "validate platform": - Installed hashicorp/google-beta v6.1.0 (signed by HashiCorp)
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #0 - "validate platform": selections it made above. Include this file in your version control repository
Step #0 - "validate platform": so that Terraform can guarantee to make the same selections by default when
Step #0 - "validate platform": you run "terraform init" in the future.
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has been successfully initialized!
Step #0 - "validate platform": 
Step #0 - "validate platform": You may now begin working with Terraform. Try running "terraform plan" to see
Step #0 - "validate platform": any changes that are required for your infrastructure. All Terraform commands
Step #0 - "validate platform": should now work.
Step #0 - "validate platform": 
Step #0 - "validate platform": If you ever set or change modules or backend configuration for Terraform,
Step #0 - "validate platform": rerun this command to reinitialize your working directory. If you forget, other
Step #0 - "validate platform": commands will detect it and remind you to do so if necessary.
Step #0 - "validate platform": Success! The configuration is valid.
Step #0 - "validate platform": 
Finished Step #0 - "validate platform"
Starting Step #3 - "validate rag"
Starting Step #2 - "validate jupyterhub"
Starting Step #1 - "validate ray"
Step #3 - "validate rag": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #1 - "validate ray": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #2 - "validate jupyterhub": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing the backend...
Step #3 - "validate rag": Initializing modules...
Step #3 - "validate rag": - cloudsql in ../../modules/cloudsql
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing the backend...
Step #1 - "validate ray": Initializing modules...
Step #1 - "validate ray": - gcs in ../../modules/gcs
Step #1 - "validate ray": - infra in ../../infrastructure
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/sql-db/google 20.0.0 for cloudsql.cloudsql...
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing the backend...
Step #2 - "validate jupyterhub": Initializing modules...
Step #2 - "validate jupyterhub": - gcs in ../../modules/gcs
Step #2 - "validate jupyterhub": - infra in ../../infrastructure
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #1 - "validate ray": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #1 - "validate ray": - infra.custom-network in ../../modules/gcp-network
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #2 - "validate jupyterhub": - infra.custom-network in ../../modules/gcp-network
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - cloudsql.cloudsql in .terraform/modules/cloudsql.cloudsql/modules/postgresql
Step #3 - "validate rag": - frontend in frontend
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for frontend.frontend-workload-identity...
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #1 - "validate ray": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - frontend.frontend-workload-identity in .terraform/modules/frontend.frontend-workload-identity/modules/workload-identity
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #1 - "validate ray": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for frontend.frontend-workload-identity.annotate-sa...
Step #1 - "validate ray": - kuberay-cluster in ../../modules/kuberay-cluster
Step #1 - "validate ray": - kuberay-cluster.iap_auth in ../../modules/iap
Step #1 - "validate ray": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-workload-identity...
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #2 - "validate jupyterhub": - jupyterhub in ../../modules/jupyter
Step #2 - "validate jupyterhub": - jupyterhub.iap_auth in ../../modules/iap
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa in .terraform/modules/frontend.frontend-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/frontend.frontend-workload-identity.annotate-sa
Step #3 - "validate rag": - frontend.iap_auth in ../../modules/iap
Step #3 - "validate rag": - gcs in ../../modules/gcs
Step #3 - "validate rag": - inference-server in ../../modules/inference-service
Step #3 - "validate rag": - infra in ../../infrastructure
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #3 - "validate rag": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #3 - "validate rag": - infra.custom-network in ../../modules/gcp-network
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - kuberay-workload-identity in .terraform/modules/kuberay-workload-identity/modules/workload-identity
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for kuberay-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #1 - "validate ray": - kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #1 - "validate ray": - kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-workload-identity.annotate-sa
Step #1 - "validate ray": - namespace in ../../modules/kubernetes-namespace
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #2 - "validate jupyterhub": - namespace in ../../modules/kubernetes-namespace
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #3 - "validate rag": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.private-gke-standard-cluster.gke...
Step #3 - "validate rag": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #3 - "validate rag": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 33.0.0 for infra.public-gke-standard-cluster.gke...
Step #3 - "validate rag": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #3 - "validate rag": - jupyterhub in ../../modules/jupyter
Step #3 - "validate rag": - jupyterhub.iap_auth in ../../modules/iap
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #3 - "validate rag": - kuberay-cluster in ../../modules/kuberay-cluster
Step #3 - "validate rag": - kuberay-cluster.iap_auth in ../../modules/iap
Step #3 - "validate rag": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-workload-identity...
Step #2 - "validate jupyterhub": - project-services in .terraform/modules/project-services/modules/project_services
Step #1 - "validate ray": - project-services in .terraform/modules/project-services/modules/project_services
Step #3 - "validate rag": - kuberay-workload-identity in .terraform/modules/kuberay-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.5.0 for kuberay-workload-identity.annotate-sa...
Step #3 - "validate rag": - kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing provider plugins...
Step #2 - "validate jupyterhub": - terraform.io/builtin/terraform is built in to Terraform
Step #2 - "validate jupyterhub": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #3 - "validate rag": - kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-workload-identity.annotate-sa
Step #3 - "validate rag": - namespace in ../../modules/kubernetes-namespace
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing provider plugins...
Step #1 - "validate ray": - terraform.io/builtin/terraform is built in to Terraform
Step #1 - "validate ray": - Finding hashicorp/time versions matching "0.11.1"...
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #2 - "validate jupyterhub": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #1 - "validate ray": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.40.0, != 5.44.0, < 6.0.0, <= 6.1.0, < 7.0.0"...
Step #1 - "validate ray": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.40.0, != 5.44.0, < 6.0.0, <= 6.1.0, < 7.0.0"...
Step #1 - "validate ray": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #1 - "validate ray": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #2 - "validate jupyterhub": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #1 - "validate ray": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #1 - "validate ray": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #2 - "validate jupyterhub": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #1 - "validate ray": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #2 - "validate jupyterhub": - Installing hashicorp/external v2.3.4...
Step #1 - "validate ray": - Installing hashicorp/random v3.6.3...
Step #3 - "validate rag": - project-services in .terraform/modules/project-services/modules/project_services
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing provider plugins...
Step #3 - "validate rag": - terraform.io/builtin/terraform is built in to Terraform
Step #3 - "validate rag": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0, ~> 3.1"...
Step #3 - "validate rag": - Finding hashicorp/null versions matching ">= 2.1.0, ~> 3.1"...
Step #3 - "validate rag": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #3 - "validate rag": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.12.0, >= 5.40.0, != 5.44.0, < 6.0.0, < 7.0.0"...
Step #2 - "validate jupyterhub": - Installed hashicorp/external v2.3.4 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/random v3.6.3 (signed by HashiCorp)
Step #3 - "validate rag": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.12.0, >= 5.40.0, != 5.44.0, < 6.0.0, <= 6.1.0, < 7.0.0"...
Step #3 - "validate rag": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #2 - "validate jupyterhub": - Installing hashicorp/null v3.2.3...
Step #1 - "validate ray": - Installing hashicorp/kubernetes v2.18.1...
Step #3 - "validate rag": - Finding hashicorp/time versions matching "0.11.1"...
Step #3 - "validate rag": - Installing hashicorp/helm v2.8.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/null v3.2.3 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/google-beta v5.45.0...
Step #1 - "validate ray": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/google v5.45.0...
Step #3 - "validate rag": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/time v0.11.1...
Step #3 - "validate rag": - Installed hashicorp/time v0.11.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/random v3.6.3...
Step #3 - "validate rag": - Installed hashicorp/random v3.6.3 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/google-beta v5.45.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/null v3.2.3...
Step #2 - "validate jupyterhub": - Installing hashicorp/helm v2.8.0...
Step #1 - "validate ray": - Installed hashicorp/google v5.45.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/external v2.3.4...
Step #3 - "validate rag": - Installed hashicorp/null v3.2.3 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/external v2.3.4...
Step #1 - "validate ray": - Installed hashicorp/external v2.3.4 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/external v2.3.4 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/null v3.2.3...
Step #3 - "validate rag": - Installing hashicorp/kubernetes v2.18.1...
Step #2 - "validate jupyterhub": - Installing hashicorp/kubernetes v2.18.1...
Step #1 - "validate ray": - Installed hashicorp/null v3.2.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/time v0.11.1...
Step #1 - "validate ray": - Installed hashicorp/time v0.11.1 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/google-beta v5.45.0...
Step #2 - "validate jupyterhub": - Installing hashicorp/google v5.45.0...
Step #3 - "validate rag": - Installing hashicorp/google v5.45.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/google v5.45.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/google v5.45.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/google-beta v5.45.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/random v3.6.3...
Step #3 - "validate rag": - Installing hashicorp/google-beta v5.45.0...
Step #1 - "validate ray": - Installing hashicorp/helm v2.8.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/random v3.6.3 (signed by HashiCorp)
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #2 - "validate jupyterhub": selections it made above. Include this file in your version control repository
Step #2 - "validate jupyterhub": so that Terraform can guarantee to make the same selections by default when
Step #2 - "validate jupyterhub": you run "terraform init" in the future.
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has been successfully initialized!
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": You may now begin working with Terraform. Try running "terraform plan" to see
Step #2 - "validate jupyterhub": any changes that are required for your infrastructure. All Terraform commands
Step #2 - "validate jupyterhub": should now work.
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": If you ever set or change modules or backend configuration for Terraform,
Step #2 - "validate jupyterhub": rerun this command to reinitialize your working 
...
[Logs truncated due to log size limitations. For full logs, see https://storage.cloud.google.com/ai-on-gke-build-logs/log-7e3e68a4-7e05-4066-9c33-44d18571eb42.txt.]
...
nup gke cluster":               - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":               - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":       - node_pool {
Step #11 - "cleanup gke cluster":           - initial_node_count          = 0 -> null
Step #11 - "cleanup gke cluster":           - instance_group_urls         = [] -> null
Step #11 - "cleanup gke cluster":           - managed_instance_group_urls = [] -> null
Step #11 - "cleanup gke cluster":           - max_pods_per_node           = 16 -> null
Step #11 - "cleanup gke cluster":           - name                        = "nap-1lhhsuvj" -> null
Step #11 - "cleanup gke cluster":           - node_count                  = 0 -> null
Step #11 - "cleanup gke cluster":           - node_locations              = [
Step #11 - "cleanup gke cluster":               - "us-east4-a",
Step #11 - "cleanup gke cluster":               - "us-east4-c",
Step #11 - "cleanup gke cluster":             ] -> null
Step #11 - "cleanup gke cluster":           - version                     = "1.30.5-gke.1699000" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - autoscaling {
Step #11 - "cleanup gke cluster":               - location_policy      = "BALANCED" -> null
Step #11 - "cleanup gke cluster":               - max_node_count       = 1000 -> null
Step #11 - "cleanup gke cluster":               - min_node_count       = 0 -> null
Step #11 - "cleanup gke cluster":               - total_max_node_count = 0 -> null
Step #11 - "cleanup gke cluster":               - total_min_node_count = 0 -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - management {
Step #11 - "cleanup gke cluster":               - auto_repair  = true -> null
Step #11 - "cleanup gke cluster":               - auto_upgrade = true -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - network_config {
Step #11 - "cleanup gke cluster":               - create_pod_range     = false -> null
Step #11 - "cleanup gke cluster":               - enable_private_nodes = false -> null
Step #11 - "cleanup gke cluster":               - pod_ipv4_cidr_block  = "10.245.0.0/17" -> null
Step #11 - "cleanup gke cluster":               - pod_range            = "gke-ml-da30a6b-864-7e3e68a4-cluster-pods-d5e4fc27" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - node_config {
Step #11 - "cleanup gke cluster":               - disk_size_gb                = 250 -> null
Step #11 - "cleanup gke cluster":               - disk_type                   = "pd-balanced" -> null
Step #11 - "cleanup gke cluster":               - effective_taints            = [
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "cloud.google.com/compute-class"
Step #11 - "cleanup gke cluster":                       - value  = "Accelerator"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "cloud.google.com/gke-ephemeral-storage-local-ssd"
Step #11 - "cleanup gke cluster":                       - value  = "true"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "nvidia.com/gpu"
Step #11 - "cleanup gke cluster":                       - value  = "present"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                 ] -> null
Step #11 - "cleanup gke cluster":               - enable_confidential_storage = false -> null
Step #11 - "cleanup gke cluster":               - image_type                  = "COS_CONTAINERD" -> null
Step #11 - "cleanup gke cluster":               - labels                      = {
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/compute-class"         = "Accelerator"
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/gke-accelerator-count" = "2"
Step #11 - "cleanup gke cluster":                 } -> null
Step #11 - "cleanup gke cluster":               - local_ssd_count             = 0 -> null
Step #11 - "cleanup gke cluster":               - logging_variant             = "DEFAULT" -> null
Step #11 - "cleanup gke cluster":               - machine_type                = "g2-standard-24" -> null
Step #11 - "cleanup gke cluster":               - metadata                    = {
Step #11 - "cleanup gke cluster":                   - "disable-legacy-endpoints" = "true"
Step #11 - "cleanup gke cluster":                 } -> null
Step #11 - "cleanup gke cluster":               - oauth_scopes                = [
Step #11 - "cleanup gke cluster":                   - "https://www.googleapis.com/auth/cloud-platform",
Step #11 - "cleanup gke cluster":                   - "https://www.googleapis.com/auth/userinfo.email",
Step #11 - "cleanup gke cluster":                 ] -> null
Step #11 - "cleanup gke cluster":               - preemptible                 = false -> null
Step #11 - "cleanup gke cluster":               - resource_labels             = {} -> null
Step #11 - "cleanup gke cluster":               - resource_manager_tags       = {} -> null
Step #11 - "cleanup gke cluster":               - service_account             = "tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":               - spot                        = false -> null
Step #11 - "cleanup gke cluster":               - tags                        = [] -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - ephemeral_storage_config {
Step #11 - "cleanup gke cluster":                   - local_ssd_count = 2 -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - guest_accelerator {
Step #11 - "cleanup gke cluster":                   - count = 2 -> null
Step #11 - "cleanup gke cluster":                   - type  = "nvidia-l4" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":                   - gpu_driver_installation_config {
Step #11 - "cleanup gke cluster":                       - gpu_driver_version = "DEFAULT" -> null
Step #11 - "cleanup gke cluster":                     }
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - gvnic {
Step #11 - "cleanup gke cluster":                   - enabled = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - reservation_affinity {
Step #11 - "cleanup gke cluster":                   - consume_reservation_type = "NO_RESERVATION" -> null
Step #11 - "cleanup gke cluster":                   - values                   = [] -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - shielded_instance_config {
Step #11 - "cleanup gke cluster":                   - enable_integrity_monitoring = true -> null
Step #11 - "cleanup gke cluster":                   - enable_secure_boot          = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - workload_metadata_config {
Step #11 - "cleanup gke cluster":                   - mode = "GKE_METADATA" -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - placement_policy {}
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - upgrade_settings {
Step #11 - "cleanup gke cluster":               - max_surge       = 1 -> null
Step #11 - "cleanup gke cluster":               - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":               - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - node_pool_defaults {
Step #11 - "cleanup gke cluster":           - node_config_defaults {
Step #11 - "cleanup gke cluster":               - logging_variant = "DEFAULT" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - gcfs_config {
Step #11 - "cleanup gke cluster":                   - enabled = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - notification_config {
Step #11 - "cleanup gke cluster":           - pubsub {
Step #11 - "cleanup gke cluster":               - enabled = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - pod_security_policy_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - private_cluster_config {
Step #11 - "cleanup gke cluster":           - enable_private_endpoint = false -> null
Step #11 - "cleanup gke cluster":           - enable_private_nodes    = false -> null
Step #11 - "cleanup gke cluster":           - private_endpoint        = "10.128.0.2" -> null
Step #11 - "cleanup gke cluster":           - public_endpoint         = "34.21.14.40" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - master_global_access_config {
Step #11 - "cleanup gke cluster":               - enabled = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - protect_config {
Step #11 - "cleanup gke cluster":           - workload_vulnerability_mode = "DISABLED" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - workload_config {
Step #11 - "cleanup gke cluster":               - audit_mode = "DISABLED" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - release_channel {
Step #11 - "cleanup gke cluster":           - channel = "REGULAR" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - secret_manager_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - security_posture_config {
Step #11 - "cleanup gke cluster":           - mode               = "DISABLED" -> null
Step #11 - "cleanup gke cluster":           - vulnerability_mode = "VULNERABILITY_DISABLED" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - service_external_ips_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - timeouts {
Step #11 - "cleanup gke cluster":           - create = "45m" -> null
Step #11 - "cleanup gke cluster":           - delete = "45m" -> null
Step #11 - "cleanup gke cluster":           - update = "45m" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - vertical_pod_autoscaling {
Step #11 - "cleanup gke cluster":           - enabled = true -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - workload_identity_config {
Step #11 - "cleanup gke cluster":           - workload_pool = "gke-ai-eco-dev.svc.id.goog" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-metric_writer" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYm/KoeAhk=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/monitoring.metricWriter/serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/monitoring.metricWriter" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-nodeService_account" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYm/KoeAhk=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/container.defaultNodeServiceAccount/serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/container.defaultNodeServiceAccount" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-resourceMetadata-writer" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYm/KoeAhk=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/stackdriver.resourceMetadata.writer/serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/stackdriver.resourceMetadata.writer" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_service_account" "cluster_service_account" {
Step #11 - "cleanup gke cluster":       - account_id   = "tf-gke-ml-da30a6b-864--cfep" -> null
Step #11 - "cleanup gke cluster":       - disabled     = false -> null
Step #11 - "cleanup gke cluster":       - display_name = "Terraform-managed service account for cluster ml-da30a6b-864-7e3e68a4-cluster" -> null
Step #11 - "cleanup gke cluster":       - email        = "tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - id           = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member       = "serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - name         = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project      = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - unique_id    = "104779122931957417123" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "random_string" "cluster_service_account_suffix" {
Step #11 - "cleanup gke cluster":       - id          = "cfep" -> null
Step #11 - "cleanup gke cluster":       - length      = 4 -> null
Step #11 - "cleanup gke cluster":       - lower       = true -> null
Step #11 - "cleanup gke cluster":       - min_lower   = 0 -> null
Step #11 - "cleanup gke cluster":       - min_numeric = 0 -> null
Step #11 - "cleanup gke cluster":       - min_special = 0 -> null
Step #11 - "cleanup gke cluster":       - min_upper   = 0 -> null
Step #11 - "cleanup gke cluster":       - number      = true -> null
Step #11 - "cleanup gke cluster":       - numeric     = true -> null
Step #11 - "cleanup gke cluster":       - result      = "cfep" -> null
Step #11 - "cleanup gke cluster":       - special     = false -> null
Step #11 - "cleanup gke cluster":       - upper       = false -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Plan: 0 to add, 0 to change, 10 to destroy.
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Changes to Outputs:
Step #11 - "cleanup gke cluster":   - ca_certificate   = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - cluster_location = "us-east4" -> null
Step #11 - "cleanup gke cluster":   - cluster_name     = "ml-da30a6b-864-7e3e68a4-cluster" -> null
Step #11 - "cleanup gke cluster":   - endpoint         = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - private_cluster  = false -> null
Step #11 - "cleanup gke cluster":   - project_id       = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0]: Destroying... [id=gke-ai-eco-dev/roles/monitoring.metricWriter/serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destroying... [id=gke-ai-eco-dev/roles/container.defaultNodeServiceAccount/serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0]: Destroying... [id=gke-ai-eco-dev/roles/stackdriver.resourceMetadata.writer/serviceAccount:tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Destroying... [id=projects/gke-ai-eco-dev/locations/us-east4/clusters/ml-da30a6b-864-7e3e68a4-cluster]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0]: Destruction complete after 9s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 3m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 3m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 3m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 3m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-da30a6b-864-7e3e68a4-cluster, 4m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Destruction complete after 4m2s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destroying... [id=projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-da30a6b-864--cfep@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix[0]: Destroying... [id=cfep]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destroying... [id=https%3A%2F%2Fwww.googleapis.com%2Fcompute%2Fv1%2Fprojects%2Fgke-ai-eco-dev%2Fglobal%2Fnetworks%2Fml-da30a6b-864-7e3e68a4-true:servicenetworking.googleapis.com]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-da30a6b-864-7e3e68a4-true"]: Destroying... [id=projects/gke-ai-eco-dev/regions/us-east4/subnetworks/ml-da30a6b-864-7e3e68a4-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destroying... [id=projects/gke-ai-eco-dev/global/addresses/google-managed-services-ml-da30a6b-864-7e3e68a4-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-da30a6b-864-7e3e68a4-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-da30a6b-864-7e3e68a4-true, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Still destroying... [id=projects/gke-ai-eco-dev/global/addresse...-services-ml-da30a6b-864-7e3e68a4-true, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destruction complete after 11s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-da30a6b-864-7e3e68a4-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-da30a6b-864-7e3e68a4-true, 20s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-da30a6b-864-7e3e68a4-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-da30a6b-864-7e3e68a4-true, 30s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-da30a6b-864-7e3e68a4-true"]: Destruction complete after 32s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-da30a6b-864-7e3e68a4-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Still destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-da30a6b-864-7e3e68a4-true, 10s elapsed]
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Error: Error waiting for Deleting Network: The network resource 'projects/gke-ai-eco-dev/global/networks/ml-da30a6b-864-7e3e68a4-true' is already being used by 'projects/gke-ai-eco-dev/global/firewalls/ml-da30a6b-864-7e3e68a4-true-e7mxb3iqxpacep2yufaua5xe'
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": 
Finished Step #11 - "cleanup gke cluster"
ERROR: step exited with non-zero status: 1
Starting Step #12 - "check result"
Step #12 - "check result": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #12 - "check result": cat: /workspace/rag_prompt_result.txt: No such file or directory
Step #12 - "check result": rag prompt test failed
Finished Step #12 - "check result"
ERROR
ERROR: build step 12 "gcr.io/gke-ai-eco-dev/terraform" failed: step exited with non-zero status: 1

Build Log: https://storage.cloud.google.com/ai-on-gke-build-logs/log-7e3e68a4-7e05-4066-9c33-44d18571eb42.txt