Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: cluster_creation failed #123744

Closed
cockroach-teamcity opened this issue May 7, 2024 · 1,354 comments
Closed

roachtest: cluster_creation failed #123744

cockroach-teamcity opened this issue May 7, 2024 · 1,354 comments
Labels
branch-master Failures and bugs on the master branch. O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 7, 2024 •
edited by cockroach-jira-scripts
Loading

roachtest.cluster_creation failed with artifacts on master @ e90d15c9807775a1a1d89f75cac4c9383c325709:

test ycsb/B/nodes=3/cpu=32/uniform failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15141267-1715060891-167-n4cpu32-0001: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPLDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 350, Current Usage: 320, Additional Required: 32, (Minimum) New Limit Required: 352. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPLDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPLDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-38522
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue labels May 7, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 7, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ e90d15c9807775a1a1d89f75cac4c9383c325709:

test zfs/ycsb/A/nodes=3/cpu=32 failed: (test_runner.go:791).func4: node creation with zfs file system not yet supported on azure [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

This was referenced May 7, 2024
Open
Open
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test costfuzz/seed-multi-region failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-70-n9cpu4-geo-0008: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_D4ds_v5' is currently not available in location 'westus2'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details." Target="vmSize" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=3/multi-region failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-93-n8cpu16-geo-0005: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_D16ds_v5' is currently not available in location 'westus2'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details." Target="vmSize" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test schemachange/leasing-benchmark failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-124-n3cpu4-geo-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_D4ds_v5' is currently not available in location 'westus2'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details." Target="vmSize" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test sqlsmith/setup=seed-multi-region/setting=multi-region failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-130-n4cpu4-geo-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_D4ds_v5' is currently not available in location 'westus2'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details." Target="vmSize" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-135-n2cpu32-0001: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPLDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 350, Current Usage: 320, Additional Required: 32, (Minimum) New Limit Required: 352. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPLDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPLDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-145-n9cpu4-geo-0007: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_D4ds_v5' is currently not available in location 'westus2'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details." Target="vmSize" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256 failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-152-n4cpu32-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPLDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 350, Current Usage: 320, Additional Required: 32, (Minimum) New Limit Required: 352. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPLDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPLDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15157871-1715147435-156-n4cpu32-0001: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPLDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 350, Current Usage: 320, Additional Required: 32, (Minimum) New Limit Required: 352. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPLDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPLDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:791).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-15157859-1715147440-141-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-05-08t18_14_57z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script4109779878 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-15157859-1715147440-141-n4cpu96-0001 teamcity-15157859-1715147440-141-n4cpu96-0002 teamcity-15157859-1715147440-141-n4cpu96-0003 teamcity-15157859-1715147440-141-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-15157859-1715147440-141-n4cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-15157859-1715147440-141-n4cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240501'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
	metric name = compute.googleapis.com/n2d_cpus
	limit name = N2D-CPUS-per-project-region
	limit = 1000.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:791).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-05-08t18_26_00z,roachprod=true,cluster=teamcity-15157859-1715147440-147-n4cpu96,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script2887523065 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-15157859-1715147440-147-n4cpu96-0001 teamcity-15157859-1715147440-147-n4cpu96-0002 teamcity-15157859-1715147440-147-n4cpu96-0003 teamcity-15157859-1715147440-147-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-15157859-1715147440-147-n4cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-15157859-1715147440-147-n4cpu96-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240501'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
	metric name = compute.googleapis.com/n2d_cpus
	limit name = N2D-CPUS-per-project-region
	limit = 1000.0
	dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=96
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:

test ledger/nodes=6/multi-az failed: (test_runner.go:791).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-15157859-1715147440-154-n7cpu16-geo,lifetime=12h0m0s,arch=amd64,created=2024-05-08t18_47_27z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2779245506 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-c teamcity-15157859-1715147440-154-n7cpu16-geo-0005 teamcity-15157859-1715147440-154-n7cpu16-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1ee6c19177041c25481e465186ce8f7294c82547:

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15175591-1715233814-38-n8cpu8-0002: Future#WaitForCompletion: context has been cancelled: StatusCode=200 -- Original Error: context deadline exceeded [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=8
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1ee6c19177041c25481e465186ce8f7294c82547:

test failover/non-system/blackhole-recv/lease=expiration failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15175591-1715233814-77-n7cpu2-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 700, Current Usage: 700, Additional Required: 2, (Minimum) New Limit Required: 702. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:702,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1ee6c19177041c25481e465186ce8f7294c82547:

test failover/non-system/blackhole-send failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15175591-1715233814-81-n7cpu2-0004: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 700, Current Usage: 700, Additional Required: 2, (Minimum) New Limit Required: 702. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:702,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1ee6c19177041c25481e465186ce8f7294c82547:

test failover/non-system/blackhole-send/lease=expiration failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15175591-1715233814-85-n7cpu2-0004: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 700, Current Usage: 700, Additional Required: 2, (Minimum) New Limit Required: 702. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:702,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1ee6c19177041c25481e465186ce8f7294c82547:

test failover/non-system/blackhole/lease=expiration failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15175591-1715233814-86-n7cpu2-0004: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 700, Current Usage: 700, Additional Required: 2, (Minimum) New Limit Required: 702. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:702,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=2
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1ee6c19177041c25481e465186ce8f7294c82547:

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15175591-1715233814-111-n8cpu16-geo-0005: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="SkuNotAvailable" Message="The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_D16ds_v5' is currently not available in location 'westus2'. Please try another size or deploy to a different location or different zone. See https://aka.ms/azureskunotavailable for details." Target="vmSize" [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 1ee6c19177041c25481e465186ce8f7294c82547:

test indexes/2/nodes=6/multi-region failed: (test_runner.go:791).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue May 9, 2024
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-08-26t12_28_26z,roachprod=true,cluster=teamcity-16614754-1724651357-147-n9cpu4-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script846892237 --project cockroach-ephemeral --boot-disk-size=32GB --zone europe-west2-c teamcity-16614754-1724651357-147-n9cpu4-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Internal error. Please try again or contact Google Support. (Code: '62095432D2594.95DC1B6.6620177B'): exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16614754-1724651357-152-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-26t12_35_20z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1404991761 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16614754-1724651357-152-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

test sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256 failed: (test_runner.go:841).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6a.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-iWV07XgWrLjxolVsiqCcSnbdFUA= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-16628464-1724737684-20-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-27T06:12:12Z},{Key=Name,Value=teamcity-16628464-1724737684-20-n4cpu96-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-16628464-1724737684-20-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-27T06:12:12Z},{Key=Name,Value=teamcity-16628464-1724737684-20-n4cpu96-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script3708409103 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3627262018 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=96
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test kv95/enc=false/nodes=3/cpu=32/size=4kb failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-iWV07XgWrLjxolVsiqCcSnbdFUA= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-08-27T06:23:25Z},{Key=Name,Value=teamcity-16628464-1724737684-30-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16628464-1724737684-30-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-08-27T06:23:25Z},{Key=Name,Value=teamcity-16628464-1724737684-30-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16628464-1724737684-30-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script1929985195 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2422863487 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:841).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=96
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test perturbation/metamorphic/partition failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test perturbation/metamorphic/restart failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test schemachange/leasing-benchmark failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.xlarge --image-id ami-0df4b2961410d4cff --key-name teamcity-runner-iWV07XgWrLjxolVsiqCcSnbdFUA= --region ap-southeast-2 --security-group-ids sg-095d6c8fccf08e948 --subnet-id subnet-0c7c1188406b8c21b --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-08-27T07:08:15Z},{Key=Name,Value=teamcity-16628464-1724737684-58-n3cpu4-geo-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16628464-1724737684-58-n3cpu4-geo},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-08-27T07:08:15Z},{Key=Name,Value=teamcity-16628464-1724737684-58-n3cpu4-geo-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16628464-1724737684-58-n3cpu4-geo},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script3882760986 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3643255958 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m6id.xlarge capacity in the Availability Zone you requested (ap-southeast-2b). Our system will be working on provisioning additional capacity. You can currently get m6id.xlarge capacity by not specifying an Availability Zone in your request or choosing ap-southeast-2a, ap-southeast-2c.: exit status 254 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=aws
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:841).func4: in provider: azure: creating VM teamcity-16628475-1724737653-80-n4cpu96-0004: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="BadRequest" Message="Cannot create a VM of size 'Standard_D4pds_v5' because this VM size only supports a CPU Architecture of 'Arm64', but an image or disk with CPU Architecture 'x64' was given. Please check that the CPU Architecture of the image or disk is compatible with that of the VM size." [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=96
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test perturbation/metamorphic/decommission failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=32
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=2
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=true
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=azure
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.cluster_creation failed with artifacts on master @ 6866fbf2dd02a298362f888ee9dc95cb19709d3f:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16628463-1724737731-153-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-27t12_11_18z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account [email protected] --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2630285480 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16628463-1724737731-153-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=4
  • ROACHTEST_runtimeAssertionsBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@renatolabs renatolabs mentioned this issue Aug 29, 2024
Merged
This was referenced Sep 7, 2024
Closed
Open
Closed
Closed
@renatolabs
Copy link
Contributor

Subsumed by #123745.

@github-project-automation github-project-automation bot moved this from Triage to Done in Test Engineering Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. O-roachtest O-robot Originated from a bot. T-testeng TestEng Team X-infra-flake the automatically generated issue was closed due to an infrastructure problem not a product issue
Projects
No open projects
Test Engineering
Status: Done
Milestone
24.1
Development

No branches or pull requests
2 participants
@cockroach-teamcity @renatolabs