Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #108859

Closed
cockroach-teamcity opened this issue Aug 16, 2023 · 251 comments
Closed

roachtest: cluster_creation failed #108859

cockroach-teamcity opened this issue Aug 16, 2023 · 251 comments
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 16, 2023 •
edited by jlinder
Loading

roachtest.cluster_creation failed with artifacts on release-22.2 @ 6bcaf00c81cdcd82245a7e95808fcb70fbb177a5:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:677).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-S7B6ZvTuxzE9iSZaMlfmiVuktXk= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11339555-1692164464-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-16T18:25:26Z},{Key=Name,Value=teamcity-11339555-1692164464-16-n4cpu96-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11339555-1692164464-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-16T18:25:26Z},{Key=Name,Value=teamcity-11339555-1692164464-16-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script3639348093 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2252567307 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-30668
@cockroach-teamcity cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 16, 2023
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Aug 16, 2023
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 6bcaf00c81cdcd82245a7e95808fcb70fbb177a5:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:677).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-S7B6ZvTuxzE9iSZaMlfmiVuktXk= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11339555-1692164464-19-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-16T18:26:55Z},{Key=Name,Value=teamcity-11339555-1692164464-19-n4cpu96-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11339555-1692164464-19-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-16T18:26:55Z},{Key=Name,Value=teamcity-11339555-1692164464-19-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script1036641318 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3887971405 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Aug 16, 2023
Closed
Closed
Open
@srosenberg srosenberg mentioned this issue Aug 21, 2023
Closed
@srosenberg srosenberg added X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue T-testeng TestEng Team and removed C-test-failure Broken test (automatically or manually discovered). release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Aug 21, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 21, 2023

cc @cockroachdb/test-eng

@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 22, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe809eac9af8a270af9e44ccc93fc570199b336d:

test kv95/enc=false/nodes=4 was skipped due to (test_runner.go:677).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.2xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-25F7RlCRoOd6D7myVzp4K-nz4PU= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-004c6ad7121a8d5a7 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11467120-1692855677-12-n5cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11467120-1692855677-12-n5cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-24T15:28:49Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11467120-1692855677-12-n5cpu8-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11467120-1692855677-12-n5cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-08-24T15:28:49Z}] --user-data file:///tmp/aws-startup-script1514198937 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1194019534 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ a3cfc62a12e650ae8cf957b66eeda3ec37177337:

test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to (test_runner.go:677).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-highcpu-96 --labels usage=roachtest,roachprod=true,cluster=teamcity-11485014-1692941981-116-n5cpu96,lifetime=12h0m0s,arch=amd64,created=2023-08-25t18_33_37z --metadata-from-file startup-script=/tmp/gce-startup-script397724128 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11485014-1692941981-116-n5cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11485014-1692941981-116-n5cpu96-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230817'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 400000.0 in region us-east1.
 - Quota 'N2_CPUS' exceeded.  Limit: 8000.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 622c35425b31fa3b7e67830a1de0072d40f364ab:

test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to (test_runner.go:677).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-highcpu-96 --labels usage=roachtest,arch=amd64,created=2023-08-31t18_12_33z,roachprod=true,cluster=teamcity-11564534-1693460472-113-n5cpu96,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3731838750 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230817'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 400000.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Sep 2, 2023
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 8247588204126b27473f1aa2f0f20534364187d0:

test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to (test_runner.go:677).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-highcpu-96 --labels usage=roachtest,roachprod=true,cluster=teamcity-11616134-1693806066-116-n5cpu96,lifetime=12h0m0s,arch=amd64,created=2023-09-04t18_24_21z --metadata-from-file startup-script=/tmp/gce-startup-script113342207 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230831'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 400000.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ af6752b2faf39115fd322e9db76421e71b76507d:

test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to (test_runner.go:677).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-highcpu-96 --labels usage=roachtest,roachprod=true,cluster=teamcity-11663807-1694065201-114-n5cpu96,lifetime=12h0m0s,arch=amd64,created=2023-09-07t18_41_04z --metadata-from-file startup-script=/tmp/gce-startup-script1343894014 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11663807-1694065201-114-n5cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11663807-1694065201-114-n5cpu96-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11663807-1694065201-114-n5cpu96-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230831'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 400000.0 in region us-east1.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 59a2f42f39e0bfd8d5140f1493c602d9dccd43eb:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:677).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-jcagdMyUbd7xSs97TelLUDbrvI4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11717221-1694410876-16-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11717221-1694410876-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-11T18:26:23Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11717221-1694410876-16-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11717221-1694410876-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-11T18:26:23Z}] --user-data file:///tmp/aws-startup-script3332555422 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1251748744 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 59a2f42f39e0bfd8d5140f1493c602d9dccd43eb:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:677).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-jcagdMyUbd7xSs97TelLUDbrvI4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-11T18:27:56Z},{Key=Name,Value=teamcity-11717221-1694410876-19-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11717221-1694410876-19-n4cpu96}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-11T18:27:56Z},{Key=Name,Value=teamcity-11717221-1694410876-19-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11717221-1694410876-19-n4cpu96}] --user-data file:///tmp/aws-startup-script1218172549 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4183358800 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 59a2f42f39e0bfd8d5140f1493c602d9dccd43eb:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:677).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-9a7kMCoF06p7Fwv3ug2Zi8-UMF8= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-09-12T18:27:06Z},{Key=Name,Value=teamcity-11734753-1694497362-16-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11734753-1694497362-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-09-12T18:27:06Z},{Key=Name,Value=teamcity-11734753-1694497362-16-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11734753-1694497362-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script3157429667 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3647966344 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 59a2f42f39e0bfd8d5140f1493c602d9dccd43eb:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:677).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-9a7kMCoF06p7Fwv3ug2Zi8-UMF8= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-12T18:28:47Z},{Key=Name,Value=teamcity-11734753-1694497362-19-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11734753-1694497362-19-n4cpu96}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-12T18:28:47Z},{Key=Name,Value=teamcity-11734753-1694497362-19-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11734753-1694497362-19-n4cpu96}] --user-data file:///tmp/aws-startup-script532455800 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3819971256 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ 04fb1f496e472427eea8503c6e16cbdc10875aca:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:686).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-uxbwOX_Z3aZin2L0LspyNx-Camc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-09-15T18:25:44Z},{Key=Name,Value=teamcity-11786690-1694756424-16-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11786690-1694756424-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-09-15T18:25:44Z},{Key=Name,Value=teamcity-11786690-1694756424-16-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11786690-1694756424-16-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script1922580973 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3691659354 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test drain-and-decommission/nodes=9 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-14417967-1710567656-104-n9cpu4,lifetime=12h0m0s,arch=amd64,created=2024-03-16t08_31_48z --metadata-from-file startup-script=/tmp/gce-startup-script1764961685 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-104-n9cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test drop/tpcc/w=100,nodes=9 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,cluster=teamcity-14417967-1710567656-108-n9cpu4,lifetime=12h0m0s,arch=amd64,created=2024-03-16t08_35_26z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2164751973 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-108-n9cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-108-n9cpu4-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-108-n9cpu4-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-108-n9cpu4-0008].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-108-n9cpu4-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test follower-reads/mixed-version/single-region was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,arch=amd64,created=2024-03-16t08_39_57z,roachprod=true,cluster=teamcity-14417967-1710567656-113-n3cpu2,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script1802329729 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-113-n3cpu2-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-113-n3cpu2-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=2 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test import/tpcc/warehouses=1000/nodes=32 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,cluster=teamcity-14417967-1710567656-131-n32cpu4,lifetime=12h0m0s,arch=amd64,created=2024-03-16t09_59_04z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script831559864 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-131-n32cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-131-n32cpu4-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-131-n32cpu4-0026].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-131-n32cpu4-0027].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test import/tpch/nodes=8 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,cluster=teamcity-14417967-1710567656-134-n8cpu4,lifetime=12h0m0s,arch=amd64,created=2024-03-16t10_03_11z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script4152186268 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test jepsen/bank-multitable/split was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,created=2024-03-16t10_31_58z,roachprod=true,cluster=teamcity-14417967-1710567656-149-n6cpu4,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script3682785235 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The service is currently unavailable.
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test jepsen/bank-multitable/majority-ring was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-14417967-1710567656-146-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-03-16t10_31_02z --metadata-from-file startup-script=/tmp/gce-startup-script351029658 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-146-n6cpu4-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test jepsen/bank-multitable/start-kill-2 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,cluster=teamcity-14417967-1710567656-156-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-03-16t10_35_19z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script197836463 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-156-n6cpu4-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test jepsen/bank-multitable/strobe-skews was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-14417967-1710567656-162-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-03-16t10_39_36z --metadata-from-file startup-script=/tmp/gce-startup-script1170428442 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test kv50/rangelookups/relocate/nodes=8 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,arch=amd64,created=2024-03-16t12_28_09z,roachprod=true,cluster=teamcity-14417967-1710567656-187-n9cpu8,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2350258599 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-187-n9cpu8-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-187-n9cpu8-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-187-n9cpu8-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-187-n9cpu8-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-187-n9cpu8-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-187-n9cpu8-0008].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-187-n9cpu8-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test kv95/enc=false/nodes=4/ssds=8/raid0 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-03-16t12_46_46z,roachprod=true,cluster=teamcity-14417967-1710567656-196-n5cpu8 --metadata-from-file startup-script=/tmp/gce-startup-script1033340488 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-196-n5cpu8-0001].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=8

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test replicate/wide was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,arch=amd64,created=2024-03-16t12_50_44z,roachprod=true,cluster=teamcity-14417967-1710567656-200-n9cpu1,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script3447718746 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0008].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-200-n9cpu1-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=1 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test rebalance/by-load/replicas was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,arch=amd64,created=2024-03-16t12_51_29z,roachprod=true,cluster=teamcity-14417967-1710567656-202-n7cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2232811172 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-202-n7cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-202-n7cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-202-n7cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-202-n7cpu4-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test kv95/enc=false/nodes=3/cpu=32/splt=0 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-custom-32-65536 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,arch=amd64,created=2024-03-16t13_00_05z,roachprod=true,cluster=teamcity-14417967-1710567656-211-n4cpu32,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4034845035 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-211-n4cpu32-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-211-n4cpu32-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on release-22.2 @ fe88d4e539e183e79a597cf2cba8fb4ffef9a65a:

test restore2TB/nodes=32 was skipped due to (test_runner.go:726).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-highmem-4 --min-cpu-platform Intel Ice Lake --labels usage=roachtest,roachprod=true,cluster=teamcity-14417967-1710567656-236-n32cpu4m,lifetime=12h0m0s,arch=amd64,created=2024-03-16t15_43_59z --metadata-from-file startup-script=/tmp/gce-startup-script1449174831 --project cockroach-ephemeral --boot-disk-size=32GB]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0008].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0009].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0010].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0011].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0012].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0013].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0014].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0015].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0016].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0017].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0018].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0019].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0020].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0021].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0022].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0023].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0024].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0025].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0026].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0027].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0028].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0029].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0031].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-14417967-1710567656-236-n32cpu4m-0032].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced Mar 18, 2024
Closed
Closed
Closed
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 29, 2024
Open
This was referenced May 2, 2024
Closed
Closed
This was referenced May 23, 2024
Closed
Closed
This was referenced Jun 14, 2024
Closed
Closed
Closed
@cockroach-teamcity cockroach-teamcity mentioned this issue Jun 26, 2024
Closed
@renatolabs
Copy link
Contributor

Branch no longer active, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
None yet
Milestone
22.2
Development

No branches or pull requests
3 participants
@srosenberg @cockroach-teamcity @renatolabs