Process completed with exit code 2.

Unable to download artifact(s): Artifact not found for name: eager-dist-ubuntu-20.04-any-profiler Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact. For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md

Process completed with exit code 2.

test_unet_trace_perf_multi_device[wormhole_b0-True-1-2-128-25.0-1650.0-False-device_params0] RuntimeError: bad_variant_access

Process completed with exit code 1.

test_unet_trace_2cq_same_io_multi_device[wormhole_b0-True-1-2-128-device_params0-False] RuntimeError: bad_variant_access

test_unet_upblock[device_params0-upblock2-32-132-20-32-1-2] AssertionError: 0.9989989668297883

The action 'Run frequent reg tests scripts' has timed out after 30 minutes.

This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.

Process completed with exit code 1.

This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.

Process completed with exit code 1.

test_run_resnet50_trace_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7faaed40eb8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7faaed308ee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7faaf0210e07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7faaf0210d0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7faaee611d06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7fab32074b5b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7fab32073df0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x512c) [0x7fab320e359c] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fab3207452a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b2) [0x7fab320de822] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fab320dd851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fab32075194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7fab320e07d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fab320dde02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fab32075194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7fab320e07d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fab320dde02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7fab320e2f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fab3207452a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133a98) [0x7fab32076a98] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7fab320e2f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fab320dd851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7fab320739e8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(

test_run_resnet50_trace_2cqs_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f3dc1618b8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f3dc1512ee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f3dc441ae07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f3dc441ad0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f3dc281bd06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f3e0627bb5b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f3e0627adf0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x512c) [0x7f3e062ea59c] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f3e0627b52a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b2) [0x7f3e062e5822] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f3e062e4851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f3e0627c194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f3e062e77d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f3e062e4e02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f3e0627c194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f3e062e77d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f3e062e4e02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f3e062e9f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f3e0627b52a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133a98) [0x7f3e0627da98] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f3e062e9f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f3e062e4851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f3e0627a9e8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so

The action 'Run frequent reg tests scripts' has timed out after 30 minutes.

Unable to locate executable file: git-lfs. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

test_unet_upblock[device_params0-upblock2-32-132-20-32-1-2] AssertionError: 0.9989989668297883

test_unet_upblock[device_params0-upblock4-16-528-80-16-1-2] AssertionError: 0.998989618643751

Process completed with exit code 1.

This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.

Process completed with exit code 1.

test_run_resnet50_trace_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f6ba94ebb8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f6ba93e5ee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f6bac2e2e07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f6bac2e2d0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f6baa6e3d06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f6bf315f92b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f6bf315ebc0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5130) [0x7f6bf31ce670] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f6bf315f2fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7f6bf31c98f3] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f6bf31c8921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f6bf315f38e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f6bf315ff64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7f6bf31cb8a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f6bf31c8ed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f6bf315f38e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f6bf315ff64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7f6bf31cb8a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f6bf31c8ed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f6bf315f38e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7f6bf31ce034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f6bf315f2fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133868) [0x7f6bf3161868] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7f6bf31ce034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f6bf31c8921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f6bf315f38e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f6bf315e7b8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(

test_run_resnet50_trace_2cqs_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7fa21dac6b8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7fa21d9c0ee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7fa2208bde07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7fa2208bdd0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7fa21ecbed06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7fa26773792b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7fa267736bc0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5130) [0x7fa2677a6670] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fa2677372fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7fa2677a18f3] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fa2677a0921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fa26773738e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fa267737f64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7fa2677a38a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fa2677a0ed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fa26773738e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fa267737f64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7fa2677a38a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fa2677a0ed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fa26773738e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7fa2677a6034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fa2677372fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133868) [0x7fa267739868] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7fa2677a6034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fa2677a0921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fa26773738e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7fa2677367b8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so

The action 'Run frequent reg tests scripts' has timed out after 30 minutes.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3637.)

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.5770800113677979, inference_time=0.002378998324275017), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.5770800113677979, inference_time=0.002378998324275017), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.5962629318237305, inference_time=0.002376541495323181), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.5962629318237305, inference_time=0.002376541495323181), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3637.)

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

Touching and printing out SMI info

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 83 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 60 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 78 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 63 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 79 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 52 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 70 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 58 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 57 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 80 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 69 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 62 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 72 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 82 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 62 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 64 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 62 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 61 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 31 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 82 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 66 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 83 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 44 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 81 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available