Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #136671

Open
cockroach-teamcity opened this issue Dec 4, 2024 · 98 comments
Open

roachtest: cluster_creation failed #136671

cockroach-teamcity opened this issue Dec 4, 2024 · 98 comments
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 4, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test backup/KMS/GCS/n3cpu4 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18024887-1733295207-35-n3cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t06_59_55z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1752342906 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-35-n3cpu4-0001 teamcity-18024887-1733295207-35-n3cpu4-0002 teamcity-18024887-1733295207-35-n3cpu4-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-45188
@cockroach-teamcity cockroach-teamcity added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels Dec 4, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Dec 4, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test acceptance/build-info failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18024887-1733295207-39-n4cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t06_59_58z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3307843951 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-39-n4cpu4-0001 teamcity-18024887-1733295207-39-n4cpu4-0002 teamcity-18024887-1733295207-39-n4cpu4-0003 teamcity-18024887-1733295207-39-n4cpu4-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test alterpk-tpcc-250 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-12-04t06_59_58z,roachprod=true,cluster=teamcity-18024887-1733295207-38-n4cpu32,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-custom-32-65536 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2528405850 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-38-n4cpu32-0001 teamcity-18024887-1733295207-38-n4cpu32-0002 teamcity-18024887-1733295207-38-n4cpu32-0003 teamcity-18024887-1733295207-38-n4cpu32-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test allocbench/nodes=7/cpu=8/kv/r=50/ops=skew failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-04t06_59_59z,roachprod=true,cluster=teamcity-18024887-1733295207-40-n8cpu8 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script641160347 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024887-1733295207-40-n8cpu8-0001 teamcity-18024887-1733295207-40-n8cpu8-0002 teamcity-18024887-1733295207-40-n8cpu8-0003 teamcity-18024887-1733295207-40-n8cpu8-0004 teamcity-18024887-1733295207-40-n8cpu8-0005 teamcity-18024887-1733295207-40-n8cpu8-0006 teamcity-18024887-1733295207-40-n8cpu8-0007 teamcity-18024887-1733295207-40-n8cpu8-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test acceptance/status-server failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-12-04t07_00_05z,roachprod=true,cluster=teamcity-18024887-1733295207-41-n4cpu4,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script555999164 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-41-n4cpu4-0001 teamcity-18024887-1733295207-41-n4cpu4-0002 teamcity-18024887-1733295207-41-n4cpu4-0003 teamcity-18024887-1733295207-41-n4cpu4-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test backup-restore/small-ranges failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-04t06_59_50z,roachprod=true,cluster=teamcity-18024887-1733295207-32-n4cpu4 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script771632978 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024887-1733295207-32-n4cpu4-0001 teamcity-18024887-1733295207-32-n4cpu4-0002 teamcity-18024887-1733295207-32-n4cpu4-0003 teamcity-18024887-1733295207-32-n4cpu4-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18024887-1733295207-32-n4cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18024887-1733295207-32-n4cpu4-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test backup/2TB/n10cpu4 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18024887-1733295207-33-n10cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t06_59_56z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2163536467 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-33-n10cpu4-0001 teamcity-18024887-1733295207-33-n10cpu4-0002 teamcity-18024887-1733295207-33-n10cpu4-0003 teamcity-18024887-1733295207-33-n10cpu4-0004 teamcity-18024887-1733295207-33-n10cpu4-0005 teamcity-18024887-1733295207-33-n10cpu4-0006 teamcity-18024887-1733295207-33-n10cpu4-0007 teamcity-18024887-1733295207-33-n10cpu4-0008 teamcity-18024887-1733295207-33-n10cpu4-0009 teamcity-18024887-1733295207-33-n10cpu4-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024887-1733295207-33-n10cpu4-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 4, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test acceptance/version-upgrade failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18024887-1733295207-37-n4cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t06_59_57z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2330567469 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-37-n4cpu4-0001 teamcity-18024887-1733295207-37-n4cpu4-0002 teamcity-18024887-1733295207-37-n4cpu4-0003 teamcity-18024887-1733295207-37-n4cpu4-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024887-1733295207-37-n4cpu4-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Dec 4, 2024
Open
Open
Open
Open
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test backup/nodeShutdown/coordinator/n4cpu4 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-18024887-1733295207-64-n4cpu4,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_07_01z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3829061365 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024887-1733295207-64-n4cpu4-0001 teamcity-18024887-1733295207-64-n4cpu4-0002 teamcity-18024887-1733295207-64-n4cpu4-0003 teamcity-18024887-1733295207-64-n4cpu4-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test c2c/disconnect failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_07_02z,roachprod=true,cluster=teamcity-18024887-1733295207-66-n7cpu4 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script527267799 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-66-n7cpu4-0001 teamcity-18024887-1733295207-66-n7cpu4-0002 teamcity-18024887-1733295207-66-n7cpu4-0003 teamcity-18024887-1733295207-66-n7cpu4-0004 teamcity-18024887-1733295207-66-n7cpu4-0005 teamcity-18024887-1733295207-66-n7cpu4-0006 teamcity-18024887-1733295207-66-n7cpu4-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test backup/mvcc-range-tombstones failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_07_03z,roachprod=true,cluster=teamcity-18024887-1733295207-63-n3cpu8 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3638742952 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024887-1733295207-63-n3cpu8-0001 teamcity-18024887-1733295207-63-n3cpu8-0002 teamcity-18024887-1733295207-63-n3cpu8-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test c2c/shutdown/src/coordinator failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-12-04t07_07_02z,roachprod=true,cluster=teamcity-18024887-1733295207-68-n9cpu8,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3557070559 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024887-1733295207-68-n9cpu8-0001 teamcity-18024887-1733295207-68-n9cpu8-0002 teamcity-18024887-1733295207-68-n9cpu8-0003 teamcity-18024887-1733295207-68-n9cpu8-0004 teamcity-18024887-1733295207-68-n9cpu8-0005 teamcity-18024887-1733295207-68-n9cpu8-0006 teamcity-18024887-1733295207-68-n9cpu8-0007 teamcity-18024887-1733295207-68-n9cpu8-0008 teamcity-18024887-1733295207-68-n9cpu8-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test c2c/initialscan/kv0 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18024887-1733295207-65-n9cpu8,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_07_02z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1667GB,auto-delete=yes --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1980730997 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024887-1733295207-65-n9cpu8-0001 teamcity-18024887-1733295207-65-n9cpu8-0002 teamcity-18024887-1733295207-65-n9cpu8-0003 teamcity-18024887-1733295207-65-n9cpu8-0004 teamcity-18024887-1733295207-65-n9cpu8-0005 teamcity-18024887-1733295207-65-n9cpu8-0006 teamcity-18024887-1733295207-65-n9cpu8-0007 teamcity-18024887-1733295207-65-n9cpu8-0008 teamcity-18024887-1733295207-65-n9cpu8-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test c2c/kv0 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_07_03z,roachprod=true,cluster=teamcity-18024887-1733295207-67-n7cpu8 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=100GB,auto-delete=yes --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script148305863 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18024887-1733295207-67-n7cpu8-0001 teamcity-18024887-1733295207-67-n7cpu8-0002 teamcity-18024887-1733295207-67-n7cpu8-0003 teamcity-18024887-1733295207-67-n7cpu8-0004 teamcity-18024887-1733295207-67-n7cpu8-0005 teamcity-18024887-1733295207-67-n7cpu8-0006 teamcity-18024887-1733295207-67-n7cpu8-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test c2c/shutdown/src/worker failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-18024887-1733295207-69-n9cpu8,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_07_03z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script60008280 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18024887-1733295207-69-n9cpu8-0001 teamcity-18024887-1733295207-69-n9cpu8-0002 teamcity-18024887-1733295207-69-n9cpu8-0003 teamcity-18024887-1733295207-69-n9cpu8-0004 teamcity-18024887-1733295207-69-n9cpu8-0005 teamcity-18024887-1733295207-69-n9cpu8-0006 teamcity-18024887-1733295207-69-n9cpu8-0007 teamcity-18024887-1733295207-69-n9cpu8-0008 teamcity-18024887-1733295207-69-n9cpu8-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18024887-1733295207-69-n9cpu8-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test c2c/shutdown/dest/coordinator failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-18024887-1733295207-70-n9cpu8,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_07_53z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1113879581 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024887-1733295207-70-n9cpu8-0001 teamcity-18024887-1733295207-70-n9cpu8-0002 teamcity-18024887-1733295207-70-n9cpu8-0003 teamcity-18024887-1733295207-70-n9cpu8-0004 teamcity-18024887-1733295207-70-n9cpu8-0005 teamcity-18024887-1733295207-70-n9cpu8-0006 teamcity-18024887-1733295207-70-n9cpu8-0007 teamcity-18024887-1733295207-70-n9cpu8-0008 teamcity-18024887-1733295207-70-n9cpu8-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 077df0b6263d6c49f4424ce745764fe6fde77e24:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18024887-1733295207-71-n9cpu8,lifetime=12h0m0s,arch=amd64,created=2024-12-04t07_08_11z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1734764712 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18024887-1733295207-71-n9cpu8-0001 teamcity-18024887-1733295207-71-n9cpu8-0002 teamcity-18024887-1733295207-71-n9cpu8-0003 teamcity-18024887-1733295207-71-n9cpu8-0004 teamcity-18024887-1733295207-71-n9cpu8-0005 teamcity-18024887-1733295207-71-n9cpu8-0006 teamcity-18024887-1733295207-71-n9cpu8-0007 teamcity-18024887-1733295207-71-n9cpu8-0008 teamcity-18024887-1733295207-71-n9cpu8-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Dec 4, 2024
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 4ee96fb74079ae5e8a803151fdda784ccaace07d:

test tpccbench/nodes=3/cpu=16/lease=expiration failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-1ORQg_nCynGrJ915bCiKvQj64_Q= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2025-01-20T07:22:22Z},{Key=Name,Value=teamcity-18548696-1737356051-26-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18548696-1737356051-26-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2025-01-20T07:22:22Z},{Key=Name,Value=teamcity-18548696-1737356051-26-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18548696-1737356051-26-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script4136213527 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping925611039 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 4ee96fb74079ae5e8a803151fdda784ccaace07d:

test ycsb/B/nodes=3 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-1ORQg_nCynGrJ915bCiKvQj64_Q= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18548696-1737356051-29-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-20T07:24:11Z},{Key=Name,Value=teamcity-18548696-1737356051-29-n4cpu8-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18548696-1737356051-29-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-20T07:24:11Z},{Key=Name,Value=teamcity-18548696-1737356051-29-n4cpu8-0001}] --user-data file:///tmp/aws-startup-script1254384681 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2060345882 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 4ee96fb74079ae5e8a803151fdda784ccaace07d:

test ycsb/F/nodes=3 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-1ORQg_nCynGrJ915bCiKvQj64_Q= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18548696-1737356051-35-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-20T07:35:45Z},{Key=Name,Value=teamcity-18548696-1737356051-35-n4cpu8-0003}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18548696-1737356051-35-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-20T07:35:45Z},{Key=Name,Value=teamcity-18548696-1737356051-35-n4cpu8-0003}] --user-data file:///tmp/aws-startup-script653026359 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1068385212 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 22, 2025
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv95/enc=false/nodes=3/mt-shared-process failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-GrYgp2xStNd5pc08OS7VyX7Ep2I= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-22T07:10:00Z},{Key=Name,Value=teamcity-18565605-1737528772-15-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18565605-1737528772-15-n4cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-22T07:10:00Z},{Key=Name,Value=teamcity-18565605-1737528772-15-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18565605-1737528772-15-n4cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2672483196 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3983014040 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Jan 22, 2025
Open
Open
Open
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-GrYgp2xStNd5pc08OS7VyX7Ep2I= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-22T07:42:13Z},{Key=Name,Value=teamcity-18565605-1737528772-30-n4cpu16-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18565605-1737528772-30-n4cpu16}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-22T07:42:13Z},{Key=Name,Value=teamcity-18565605-1737528772-30-n4cpu16-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18565605-1737528772-30-n4cpu16}] --user-data file:///tmp/aws-startup-script4189146766 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping105535198 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test cdc/tpcc-1000/sink=cloudstorage failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2025-01-22t08_42_09z,roachprod=true,cluster=teamcity-18565604-1737528797-36-n4cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script107617915 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-18565604-1737528797-36-n4cpu16-0001 teamcity-18565604-1737528797-36-n4cpu16-0002 teamcity-18565604-1737528797-36-n4cpu16-0003 teamcity-18565604-1737528797-36-n4cpu16-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18565604-1737528797-36-n4cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18565604-1737528797-36-n4cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-18565604-1737528797-36-n4cpu16-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20250112'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
	metric name = compute.googleapis.com/t2a_cpus
	limit name = T2A-CPUS-per-project-region
	limit = 600.0
	dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test import/tpch/nodes=8 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18565604-1737528797-51-n8cpu4,lifetime=12h0m0s,arch=amd64,created=2025-01-22t09_06_29z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script77456498 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-18565604-1737528797-51-n8cpu4-0001 teamcity-18565604-1737528797-51-n8cpu4-0002 teamcity-18565604-1737528797-51-n8cpu4-0003 teamcity-18565604-1737528797-51-n8cpu4-0004 teamcity-18565604-1737528797-51-n8cpu4-0005 teamcity-18565604-1737528797-51-n8cpu4-0006 teamcity-18565604-1737528797-51-n8cpu4-0007 teamcity-18565604-1737528797-51-n8cpu4-0008]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18565604-1737528797-51-n8cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18565604-1737528797-51-n8cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-18565604-1737528797-51-n8cpu4-0008].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20250112'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test indexes/2/nodes=6/multi-region failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2025-01-22t09_06_39z,roachprod=true,cluster=teamcity-18565604-1737528797-52-n7cpu16-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script891646236 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-18565604-1737528797-52-n7cpu16-geo-0001 teamcity-18565604-1737528797-52-n7cpu16-geo-0002 teamcity-18565604-1737528797-52-n7cpu16-geo-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18565604-1737528797-52-n7cpu16-geo-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-18565604-1737528797-52-n7cpu16-geo-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20250112'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test jepsen/bank-multitable/parts-start-kill-2 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2025-01-22t09_10_35z,roachprod=true,cluster=teamcity-18565604-1737528797-56-n6cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --metadata-from-file startup-script=/tmp/gce-startup-script164335633 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-18565604-1737528797-56-n6cpu4-0001 teamcity-18565604-1737528797-56-n6cpu4-0002 teamcity-18565604-1737528797-56-n6cpu4-0003 teamcity-18565604-1737528797-56-n6cpu4-0004 teamcity-18565604-1737528797-56-n6cpu4-0005 teamcity-18565604-1737528797-56-n6cpu4-0006]
Output: WARNING: Some requests generated warnings:
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
	metric name = compute.googleapis.com/t2a_cpus
	limit name = T2A-CPUS-per-project-region
	limit = 600.0
	dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test jepsen/bank-multitable/start-kill-2 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-18565604-1737528797-60-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2025-01-22t09_14_29z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3755728470 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-18565604-1737528797-60-n6cpu4-0001 teamcity-18565604-1737528797-60-n6cpu4-0002 teamcity-18565604-1737528797-60-n6cpu4-0003 teamcity-18565604-1737528797-60-n6cpu4-0004 teamcity-18565604-1737528797-60-n6cpu4-0005 teamcity-18565604-1737528797-60-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18565604-1737528797-60-n6cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18565604-1737528797-60-n6cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-18565604-1737528797-60-n6cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20250112'.
 - You are creating a global DNS VM. VM instances using global DNS are vulnerable to cross-regional outages. To reduce the risk of widespread service disruption, use zonal DNS instead. Learn more at https://cloud.google.com/compute/docs/networking/zonal-dns
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
	metric name = compute.googleapis.com/regional_in_use_addresses
	limit name = IN-USE-ADDRESSES-per-project-region
	limit = 1024.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv0/enc=false/nodes=3/splt=0 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-18580455-1737615133-06-n4cpu8-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-06-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:06:23Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-18580455-1737615133-06-n4cpu8-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-06-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:06:23Z}] --user-data file:///tmp/aws-startup-script284052134 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4100343990 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv0/enc=true/nodes=1 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-09-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:08:52Z},{Key=Name,Value=teamcity-18580455-1737615133-09-n2cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-09-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:08:52Z},{Key=Name,Value=teamcity-18580455-1737615133-09-n2cpu8-0002}] --user-data file:///tmp/aws-startup-script2370166055 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping766366871 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv0/enc=true/nodes=3 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-12-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:11:25Z},{Key=Name,Value=teamcity-18580455-1737615133-12-n4cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-12-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:11:25Z},{Key=Name,Value=teamcity-18580455-1737615133-12-n4cpu8-0002}] --user-data file:///tmp/aws-startup-script150696529 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1495057775 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv95/enc=false/nodes=3/mt-shared-process failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:16:26Z},{Key=Name,Value=teamcity-18580455-1737615133-17-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-17-n4cpu8}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:16:26Z},{Key=Name,Value=teamcity-18580455-1737615133-17-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-17-n4cpu8}] --user-data file:///tmp/aws-startup-script2774646865 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping440058460 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv95/enc=false/nodes=3/size=4kb failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:19:04Z},{Key=Name,Value=teamcity-18580455-1737615133-20-n4cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-20-n4cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:19:04Z},{Key=Name,Value=teamcity-18580455-1737615133-20-n4cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-20-n4cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1970209230 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping43772353 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv95/enc=false/nodes=3/size=64kb failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-23-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:21:41Z},{Key=Name,Value=teamcity-18580455-1737615133-23-n4cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-23-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:21:41Z},{Key=Name,Value=teamcity-18580455-1737615133-23-n4cpu8-0002}] --user-data file:///tmp/aws-startup-script1746953381 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3247969224 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test kv95limited-spans/enc=false/nodes=1/splt=0/seq/no-load-splitting failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-26-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:24:17Z},{Key=Name,Value=teamcity-18580455-1737615133-26-n2cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-26-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:24:17Z},{Key=Name,Value=teamcity-18580455-1737615133-26-n2cpu8-0002}] --user-data file:///tmp/aws-startup-script1252928395 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1531443529 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test tpccbench/nodes=3/cpu=16 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-18580455-1737615133-39-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:40:24Z},{Key=Name,Value=teamcity-18580455-1737615133-39-n4cpu16-0001},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-18580455-1737615133-39-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-23T07:40:24Z},{Key=Name,Value=teamcity-18580455-1737615133-39-n4cpu16-0001},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script3986708200 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1030843209 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 0cfe910eb6e254e6df5ad00f5fb0ade1eae9a9c3:

test tpccbench/nodes=3/cpu=16/enc=true failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-mdv3TTH-JGB-5eNA29KzDTjutUY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2025-01-23T07:43:15Z},{Key=Name,Value=teamcity-18580455-1737615133-42-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-42-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2025-01-23T07:43:15Z},{Key=Name,Value=teamcity-18580455-1737615133-42-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18580455-1737615133-42-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script3465137016 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping290328523 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 4030c7cebbc22332a08a20a20598099df9e0fd13:

test kv95/enc=false/nodes=3/size=4kb failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-w7tuBhBYOeN43qwGfA5VHKOLDuQ= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-24T07:08:00Z},{Key=Name,Value=teamcity-18593846-1737701585-14-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18593846-1737701585-14-n4cpu8}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-24T07:08:00Z},{Key=Name,Value=teamcity-18593846-1737701585-14-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18593846-1737701585-14-n4cpu8}] --user-data file:///tmp/aws-startup-script946314335 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1547625425 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ 4030c7cebbc22332a08a20a20598099df9e0fd13:

test kv95/enc=false/nodes=3/size=64kb failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-w7tuBhBYOeN43qwGfA5VHKOLDuQ= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-18593846-1737701585-17-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-24T07:10:29Z},{Key=Name,Value=teamcity-18593846-1737701585-17-n4cpu8-0003},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-18593846-1737701585-17-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-24T07:10:29Z},{Key=Name,Value=teamcity-18593846-1737701585-17-n4cpu8-0003},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script118205588 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3874827116 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ d6a6abad497880d98f8258ab4306a3506e96e27e:

test kv95/enc=false/nodes=3/tracing failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-9fIMScoFLoOSCMT1fz5I51oxs9g= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-18609089-1737788031-17-n4cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18609089-1737788031-17-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-25T07:12:56Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-18609089-1737788031-17-n4cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18609089-1737788031-17-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-25T07:12:56Z}] --user-data file:///tmp/aws-startup-script964101878 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1327785499 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ d6a6abad497880d98f8258ab4306a3506e96e27e:

test ycsb/B/nodes=3 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-ssPluMATKTpayMQfQze0sJW592M= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-26T07:20:02Z},{Key=Name,Value=teamcity-18613997-1737874282-21-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-21-n4cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-26T07:20:02Z},{Key=Name,Value=teamcity-18613997-1737874282-21-n4cpu8-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-21-n4cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script3165745140 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1280893167 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ d6a6abad497880d98f8258ab4306a3506e96e27e:

test ycsb/D/nodes=3 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-ssPluMATKTpayMQfQze0sJW592M= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-24-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-26T07:22:32Z},{Key=Name,Value=teamcity-18613997-1737874282-24-n4cpu8-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-24-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-26T07:22:32Z},{Key=Name,Value=teamcity-18613997-1737874282-24-n4cpu8-0002}] --user-data file:///tmp/aws-startup-script3402502425 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4200127031 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ d6a6abad497880d98f8258ab4306a3506e96e27e:

test ycsb/F/nodes=3 failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-ssPluMATKTpayMQfQze0sJW592M= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2025-01-26T07:25:02Z},{Key=Name,Value=teamcity-18613997-1737874282-27-n4cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-27-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2025-01-26T07:25:02Z},{Key=Name,Value=teamcity-18613997-1737874282-27-n4cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-27-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script3233175587 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4121091151 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.2 @ d6a6abad497880d98f8258ab4306a3506e96e27e:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: (test_runner.go:752).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-ssPluMATKTpayMQfQze0sJW592M= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-26T07:46:33Z},{Key=Name,Value=teamcity-18613997-1737874282-35-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-35-n4cpu16},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2025-01-26T07:46:33Z},{Key=Name,Value=teamcity-18613997-1737874282-35-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-18613997-1737874282-35-n4cpu16},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1369386048 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3580603236 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
None yet
Milestone
23.2
Development

No branches or pull requests
1 participant
@cockroach-teamcity