The job running on runner tt-metal-ci-vm-117 has exceeded the maximum execution time of 720 minutes.

The operation was canceled.

This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.

Process completed with exit code 1.

Disk usage is 50 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available