The self-hosted runner: tt-metal-ci-vm-169 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The job running on runner tt-metal-ci-vm-62 has exceeded the maximum execution time of 720 minutes.

This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.

Process completed with exit code 1.

The operation was canceled.

The job running on runner tt-metal-ci-vm-142 has exceeded the maximum execution time of 720 minutes.

This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.

Process completed with exit code 1.

The operation was canceled.

Touching and printing out SMI info

Disk usage is 55 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 48 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 79 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available