Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #109155

Closed
cockroach-teamcity opened this issue Aug 21, 2023 · 7 comments
Closed

roachtest: cluster_creation failed #109155

cockroach-teamcity opened this issue Aug 21, 2023 · 7 comments
Labels
O-roachtest O-robot Originated from a bot. T-testeng TestEng Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 21, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on release-23.1.9-rc @ a0eff4730a6500f3445ae41a3e47b34c28cf3278:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11410901-1692596581-149-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-21t17_23_08z --metadata-from-file startup-script=/tmp/gce-startup-script2548401418 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11410901-1692596581-149-n9cpu8-geo-0004 teamcity-11410901-1692596581-149-n9cpu8-geo-0005 teamcity-11410901-1692596581-149-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-30806
@cockroach-teamcity cockroach-teamcity added branch-release-23.1.9-rc C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 21, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Aug 21, 2023
This was referenced Aug 21, 2023
Open
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.9-rc @ 23a2985ddfccf84c0d63a4c0d249cf010a8b14c8:

test ledger/nodes=6/multi-az was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-11429823-1692682980-136-n7cpu16-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-22t17_14_08z --metadata-from-file startup-script=/tmp/gce-startup-script704285149 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-b teamcity-11429823-1692682980-136-n7cpu16-geo-0003 teamcity-11429823-1692682980-136-n7cpu16-geo-0004]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230817'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 99568.0 in region us-central1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 99568.0
	dimensions = region: us-central1
vm_family: N2
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.9-rc @ 23a2985ddfccf84c0d63a4c0d249cf010a8b14c8:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-08-22t17_21_01z,roachprod=true,cluster=teamcity-11429823-1692682980-144-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script3830374982 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11429823-1692682980-144-n9cpu8-geo-0004 teamcity-11429823-1692682980-144-n9cpu8-geo-0005 teamcity-11429823-1692682980-144-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rail
Copy link
Member

rail commented Aug 22, 2023

us-west-2b is not a valid GCP zone. Should we backport #100738?
cc @benbardin

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.9-rc @ 23a2985ddfccf84c0d63a4c0d249cf010a8b14c8:

test tpccbench/nodes=6/cpu=16/multi-az was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --labels usage=roachtest,roachprod=true,cluster=teamcity-11429823-1692682980-161-n7cpu16-geo,lifetime=12h0m0s,arch=amd64,created=2023-08-22t17_55_48z --metadata-from-file startup-script=/tmp/gce-startup-script1096203699 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-f teamcity-11429823-1692682980-161-n7cpu16-geo-0001 teamcity-11429823-1692682980-161-n7cpu16-geo-0002 teamcity-11429823-1692682980-161-n7cpu16-geo-0007]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230817'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 99568.0 in region us-central1.
	metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
	limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
	limit = 99568.0
	dimensions = region: us-central1
vm_family: N2
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@lunevalex lunevalex added the T-testeng TestEng Team label Aug 23, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 23, 2023

cc @cockroachdb/test-eng

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-23.1.9-rc @ 895bc643db6c483a9a0c3089b95fe839d86b1257:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:725).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-08-23t17_42_54z,roachprod=true,cluster=teamcity-11449084-1692769421-145-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4030471960 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-11449084-1692769421-145-n9cpu8-geo-0004 teamcity-11449084-1692769421-145-n9cpu8-geo-0005 teamcity-11449084-1692769421-145-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@srosenberg srosenberg removed C-test-failure Broken test (automatically or manually discovered). release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 23, 2023
@srosenberg
Copy link
Member

superseded by #108653

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
O-roachtest O-robot Originated from a bot. T-testeng TestEng Team
Projects
No open projects
Test Engineering
Status: Done
Milestone
23.1
Development

No branches or pull requests
4 participants
@rail @srosenberg @cockroach-teamcity @lunevalex