Value cannot be null. (Parameter 'ContainerId')

Process completed with exit code 1.

FailFast: cancelling since parallel instance has failed

The operation was canceled.

FailFast: cancelling since parallel instance has failed

The operation was canceled.

FailFast: cancelling since parallel instance has failed

Process completed with exit code 1.

The operation was canceled.

FailFast: cancelling since parallel instance has failed

The operation was canceled.

Process completed with exit code 1.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x557f9e5cafd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x557f9e5cafd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x557f9e5cafd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x557f9e5cafd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x557f9e5cafd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x557f9e5c6da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x557f9e5cd078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x557f9e5c6da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x557f9e5cd078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x557f9e5c6da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x557f9e5cd078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x557f9e5c6da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x557f9e5cd078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x557f9e621fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd181049e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd18399b9cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x557f9e5c6da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x557f9e5cd078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd181188e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd17fee2083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x557f9e5c677e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x56373601dfd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x56373601dfd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x56373601dfd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x56373601dfd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x86fd9) [0x56373601dfd9] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x563736019da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x563736020078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x563736019da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x563736020078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x563736019da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x563736020078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x563736019da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x563736020078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_THROW @ /work/tt_metal/distributed/mesh_device.cpp:593: tt::exception info: command_queue() is not supported on MeshDevice - use individual devices instead backtrace: --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0xddfb8) [0x563736074fb8] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x3ace6d) [0x7fd7fbb64e6d] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0x280f9cd) [0x7fd7fe4b69cd] --- tt::tt_metal::distributed::MeshDevice::push_work(std::__1::function<void ()>, bool) --- ttnn::write_buffer(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor&, std::__1::vector<std::__1::shared_ptr<void>, std::__1::allocator<std::__1::shared_ptr<void>>>, std::__1::optional<tt::tt_metal::v0::BufferRegion> const&) --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x82da7) [0x563736019da7] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x89078) [0x563736020078] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fd7fbca3e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd7fa9fd083] --- ./build/test/ttnn/unit_tests_ttnn_tensor(+0x8277e) [0x56373601977e] " thrown in the test body.

unknown file C++ exception with description "TT_FATAL @ /work/ttnn/cpp/ttnn/tensor/tensor.hpp:254: this->mesh_device_.has_value() info: Tensor is not a mesh tensor backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x45498) [0x56391b27e498] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_single_device<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe82a3a) [0x7f9c62ebca3a] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe823cc) [0x7f9c62ebc3cc] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe820a1) [0x7f9c62ebc0a1] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(_ZN4ttnn10operations6binary15BinaryOperationILNS1_12BinaryOpTypeE0EE6invokeERKN2tt8tt_metal6TensorES9_RKNSt3__18optionalIKNS6_8DataTypeEEERKNSB_INS6_12MemoryConfigEEERKNSB_IS7_EERKNSB_INSA_6vectorINS0_5unary14UnaryWithParamENSA_9allocatorISQ_EEEEEERKNSB_ISQ_EE+0x7b) [0x7f9c62ea040b] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f748) [0x56391b288748] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f530) [0x56391b288530] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x2109d) [0x56391b25a09d] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7f9c62036e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f9c60d90083] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x1f70e) [0x56391b25870e] " thrown in the test body.

unknown file C++ exception with description "TT_FATAL @ /work/ttnn/cpp/ttnn/tensor/tensor.hpp:254: this->mesh_device_.has_value() info: Tensor is not a mesh tensor backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x45498) [0x557f79691498] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_single_device<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe82a3a) [0x7fe23397ca3a] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe823cc) [0x7fe23397c3cc] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe820a1) [0x7fe23397c0a1] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(_ZN4ttnn10operations6binary15BinaryOperationILNS1_12BinaryOpTypeE0EE6invokeERKN2tt8tt_metal6TensorES9_RKNSt3__18optionalIKNS6_8DataTypeEEERKNSB_INS6_12MemoryConfigEEERKNSB_IS7_EERKNSB_INSA_6vectorINS0_5unary14UnaryWithParamENSA_9allocatorISQ_EEEEEERKNSB_ISQ_EE+0x7b) [0x7fe23396040b] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f748) [0x557f7969b748] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f530) [0x557f7969b530] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x2109d) [0x557f7966d09d] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7fe232af6e7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fe231850083] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x1f70e) [0x557f7966b70e] " thrown in the test body.

test_deallocate[w=64-h=32] Failed: DID NOT RAISE <class 'RuntimeError'>

test_deallocate[w=64-h=32] Failed: DID NOT RAISE <class 'RuntimeError'>

test_all_gather_on_n300_post_commit[silicon_arch_name=wormhole_b0-silicon_arch_wormhole_b0=True-enable_async=True-num_iters=1-mem_config=MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::L1,shard_spec=std::nullopt)-input_dtype=DataType.BFLOAT8_B-num_devices=2-num_links=1-output_shape=[1, 1, 32, 704]-dim=3-layout=Layout.TILE] RuntimeError: TT_FATAL @ /work/ttnn/cpp/ttnn/operations/ccl/all_gather/device/all_gather_op.cpp:242: num_devices > 1 info: all_gather op will only work for num_devices > 1, but has 1 backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xbd4820) [0x7f8806d4d820] --- ttnn::operations::ccl::all_gather(tt::tt_metal::Tensor const&, int, unsigned int, std::__1::optional<tt::tt_metal::MemoryConfig> const&, std::__1::optional<unsigned long>, std::__1::optional<unsigned long>, ttnn::ccl::Topology) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x22a0e86) [0x7f8808419e86] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x22a0d27) [0x7f8808419d27] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x22a09f5) [0x7f88084199f5] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x22a090e) [0x7f880841990e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xbf97ae) [0x7f8806d727ae] --- /usr/bin/python3(PyCFunction_Call+0x59) [0x5f59b9] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3() [0x50adc8] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3() [0x59c85c] --- /usr/bin/python3(PyObject_Call+0x27e) [0x5f534e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3() [0x59c54f] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x62f5) [0x570d65] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1882) [0x56c2f2] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1882) [0x56c2f2] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3() [0x50a59e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3() [0x59c54f] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x62f5) [0x570d65] --- /usr/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5d66] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x859) [0x56b2c9] --- /usr/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5d66] --- /usr/bin/python3

test_create_min_width_shard[True-8-1-128-1] assert 0 == 1 + where 0 = <bound method PyCapsule.num_program_cache_entries of MeshDevice(1x1 grid, 1 devices)>() + where <bound method PyCapsule.num_program_cache_entries of MeshDevice(1x1 grid, 1 devices)> = MeshDevice(1x1 grid, 1 devices).num_program_cache_entries

test_tensor_preallocation_and_write_apis[silicon_arch_name=wormhole_b0-silicon_arch_wormhole_b0=True-mesh_device=4-enable_async=False-tensor_layout=Layout.TILE-memory_location=BufferType.L1-mem_layout=TensorMemoryLayout.INTERLEAVED-in_dtype=DataType.BFLOAT16-shape=(1, 10, 64, 96)] RuntimeError: TT_THROW @ /work/ttnn/cpp/ttnn/tensor/tensor.cpp:183: tt::exception info: Not implemented backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xb65428) [0x7f07eb3c6428] --- tt::tt_metal::Tensor::Tensor(std::__1::vector<tt::tt_metal::v0::IDevice*, std::__1::allocator<tt::tt_metal::v0::IDevice*>> const&) --- tt::tt_metal::allocate_tensor_on_devices(tt::tt_metal::TensorSpec const&, std::__1::vector<tt::tt_metal::v0::IDevice*, std::__1::allocator<tt::tt_metal::v0::IDevice*>> const&) --- ttnn::operations::core::allocate_tensor_on_device(tt::tt_metal::Shape const&, tt::tt_metal::DataType, tt::tt_metal::Layout, tt::tt_metal::distributed::MeshDevice*, std::__1::optional<tt::tt_metal::MemoryConfig> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x2466929) [0x7f07eccc7929] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x24666fa) [0x7f07eccc76fa] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x24665de) [0x7f07eccc75de] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xbf97ae) [0x7f07eb45a7ae] --- /usr/bin/python3(PyCFunction_Call+0x59) [0x5f59b9] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x656f) [0x570fdf] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3() [0x59c4ce] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5e14) [0x570884] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3() [0x50a59e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3() [0x59c54f] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x62f5) [0x570d65] --- /usr/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5d66] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x859) [0x56b2c9] --- /usr/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5d66] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3() [0x50a59e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3() [0x59c54f] --- /usr/bin/python3(PyObject_Call+0x27e) [0x5f534e] --- /usr/bin/python3(_PyEva

Process completed with exit code 8.

test_run_prefetcher_post_commit[device_params={'trace_region_size': 23887872}-num_reader_cores=2-num_tensors=2-input_shapes=[(256, 512), (256, 512)]-dtypes=[<DataType.BFLOAT4_B: 4>, <DataType.BFLOAT4_B: 4>]-num_layers=5] RuntimeError: TT_THROW @ /work/ttnn/cpp/ttnn/operations/global_cb_utils.hpp:36: tt::exception info: Error finding a device for a GlobalCB in MultiDeviceGlobalCircularBuffer backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xb65428) [0x7fe5f77f3428] --- ttnn::operations::dram_prefetcher::DramPrefetcher::validate(std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&) const --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(_ZN4ttnn16device_operation6detail21launch_on_mesh_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISC_EENSD_13tensor_args_tESC_PNS4_11distributed10MeshDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0xcd) [0x7fe5f94a3bad] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x86) [0x7fe5f94a39d6] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x281590a) [0x7fe5f94a390a] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7fe5f94a32c7] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x2814c6b) [0x7fe5f94a2c6b] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x281494b) [0x7fe5f94a294b] --- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x125c8e5) [0x7fe5f7eea8e5] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x125c50c) [0x7fe5f7eea50c] --- void tt::tt_metal::operation::launch_op_func<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(std::__1::function<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> (std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<s

test_dopout assert False + where False = <function allclose at 0x7f8cfc0ec820>(0.2043994140625, 0.2, rtol=0.02) + where <function allclose at 0x7f8cfc0ec820> = np.allclose

test_run_prefetcher_post_commit[device_params={'trace_region_size': 23887872}-num_reader_cores=2-num_tensors=2-input_shapes=[(256, 512), (256, 512)]-dtypes=[<DataType.BFLOAT4_B: 4>, <DataType.BFLOAT4_B: 4>]-num_layers=5] RuntimeError: TT_THROW @ /work/ttnn/cpp/ttnn/operations/global_cb_utils.hpp:36: tt::exception info: Error finding a device for a GlobalCB in MultiDeviceGlobalCircularBuffer backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xb65428) [0x7f14f0883428] --- ttnn::operations::dram_prefetcher::DramPrefetcher::validate(std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&) const --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(_ZN4ttnn16device_operation6detail21launch_on_mesh_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISC_EENSD_13tensor_args_tESC_PNS4_11distributed10MeshDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0xcd) [0x7f14f2533bad] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x86) [0x7f14f25339d6] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x281590a) [0x7f14f253390a] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7f14f25332c7] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x2814c6b) [0x7f14f2532c6b] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x281494b) [0x7f14f253294b] --- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x125c8e5) [0x7f14f0f7a8e5] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x125c50c) [0x7f14f0f7a50c] --- void tt::tt_metal::operation::launch_op_func<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(std::__1::function<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> (std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<s

test_eq[output_dtype=DataType.BFLOAT16-w=32-h=32] AssertionError: assert 0 == (0 - 1) + where 0 = len([]) + and 0 = len([]) + where [] = <built-in method get_buffer_pages of PyCapsule object at 0x7feece511cf0>() + where <built-in method get_buffer_pages of PyCapsule object at 0x7feece511cf0> = <module 'ttnn._ttnn.reports'>.get_buffer_pages + where <module 'ttnn._ttnn.reports'> = <module 'ttnn._ttnn' from '/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so'>.reports + where <module 'ttnn._ttnn' from '/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so'> = ttnn._ttnn

test_eq[output_dtype=DataType.BFLOAT16-w=32-h=32] AssertionError: assert 0 == (0 - 1) + where 0 = len([]) + and 0 = len([]) + where [] = <built-in method get_buffer_pages of PyCapsule object at 0x7f364ca0cc90>() + where <built-in method get_buffer_pages of PyCapsule object at 0x7f364ca0cc90> = <module 'ttnn._ttnn.reports'>.get_buffer_pages + where <module 'ttnn._ttnn.reports'> = <module 'ttnn._ttnn' from '/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so'>.reports + where <module 'ttnn._ttnn' from '/home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so'> = ttnn._ttnn

Process completed with exit code 2.

Process completed with exit code 2.

unknown file C++ exception with description "TT_FATAL @ /work/ttnn/cpp/ttnn/tensor/tensor.hpp:254: this->mesh_device_.has_value() info: Tensor is not a mesh tensor backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x45498) [0x55a12aa0e498] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_single_device<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe82a3a) [0x7f82d89a2a3a] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe823cc) [0x7f82d89a23cc] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe820a1) [0x7f82d89a20a1] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(_ZN4ttnn10operations6binary15BinaryOperationILNS1_12BinaryOpTypeE0EE6invokeERKN2tt8tt_metal6TensorES9_RKNSt3__18optionalIKNS6_8DataTypeEEERKNSB_INS6_12MemoryConfigEEERKNSB_IS7_EERKNSB_INSA_6vectorINS0_5unary14UnaryWithParamENSA_9allocatorISQ_EEEEEERKNSB_ISQ_EE+0x7b) [0x7f82d898640b] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f748) [0x55a12aa18748] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f530) [0x55a12aa18530] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x2109d) [0x55a12a9ea09d] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7f82d7b1ce7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f82d6876083] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x1f70e) [0x55a12a9e870e] " thrown in the test body.

unknown file C++ exception with description "TT_FATAL @ /work/ttnn/cpp/ttnn/tensor/tensor.hpp:254: this->mesh_device_.has_value() info: Tensor is not a mesh tensor backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x45498) [0x560d8c336498] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::launch_on_single_device<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe82a3a) [0x7f2485291a3a] --- ttnn::operations::binary::BinaryDeviceOperation::tensor_return_value_t ttnn::device_operation::detail::invoke<ttnn::operations::binary::BinaryDeviceOperation>(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, ttnn::operations::binary::BinaryDeviceOperation::operation_attributes_t const&, ttnn::operations::binary::BinaryDeviceOperation::tensor_args_t const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe823cc) [0x7f24852913cc] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(+0xe820a1) [0x7f24852910a1] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/_ttnn.so(_ZN4ttnn10operations6binary15BinaryOperationILNS1_12BinaryOpTypeE0EE6invokeERKN2tt8tt_metal6TensorES9_RKNSt3__18optionalIKNS6_8DataTypeEEERKNSB_INS6_12MemoryConfigEEERKNSB_IS7_EERKNSB_INSA_6vectorINS0_5unary14UnaryWithParamENSA_9allocatorISQ_EEEEEERKNSB_ISQ_EE+0x7b) [0x7f248527540b] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f748) [0x560d8c340748] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x4f530) [0x560d8c340530] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x2109d) [0x560d8c31209d] --- void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) --- testing::Test::Run() --- testing::TestInfo::Run() --- testing::TestSuite::Run() --- testing::internal::UnitTestImpl::RunAllTests() --- bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) --- testing::UnitTest::Run() --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libgmock_main.so.1.13.0(main+0x3e) [0x7f248440be7e] --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f2483165083] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/test/tt_eager/tensors/test_async_tensor_apis(+0x1f70e) [0x560d8c31070e] " thrown in the test body.

Process completed with exit code 1.

The action has timed out.

Unexpected input(s) 'fetch-refs', valid inputs are ['repository', 'ref', 'token', 'ssh-key', 'ssh-known-hosts', 'ssh-strict', 'ssh-user', 'persist-credentials', 'path', 'clean', 'filter', 'sparse-checkout', 'sparse-checkout-cone-mode', 'fetch-depth', 'fetch-tags', 'show-progress', 'lfs', 'submodules', 'set-safe-directory', 'github-server-url']

Unexpected input(s) 'fetch-refs', valid inputs are ['repository', 'ref', 'token', 'ssh-key', 'ssh-known-hosts', 'ssh-strict', 'ssh-user', 'persist-credentials', 'path', 'clean', 'filter', 'sparse-checkout', 'sparse-checkout-cone-mode', 'fetch-depth', 'fetch-tags', 'show-progress', 'lfs', 'submodules', 'set-safe-directory', 'github-server-url']

Disk usage is high. Will try pruning stuff.

Hugepages service not found. Using old rc.local method

The Ubuntu-20.04 brownout takes place from 2025-02-01. For more details, see https://github.com/actions/runner-images/issues/11101

No files were found with the provided path: ~/run-log/20250228115854_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228115913_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228120000_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228120028_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228120121_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228120122_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228120147_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228120302_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

Disk usage is high. Will try pruning stuff.

Disk usage is high. Will try pruning stuff.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

No files were found with the provided path: ~/run-log/20250228121422_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: ~/run-log/20250228121831_sys_logs.tar. No artifacts will be uploaded.

No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.

Disk usage is 53 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 78 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 46 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 45 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 54 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 85 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 82 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 20 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 21 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 31 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 79 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 76 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 55 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 73 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 83 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 76 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 73 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 12 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 20 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 76 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 79 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 82 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 69 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 82 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 78 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 63 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 48 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 77 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 80 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 21 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 72 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 86 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 63 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 54 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 37 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 75 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 60 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 68 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 77 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 81 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 60 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 19 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 22 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 76 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 66 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 84 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 76 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 78 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 47 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 83 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 62 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 12 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 14 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 12 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 53 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 81 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 82 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 68 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 76 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 19 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 88 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 21 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 46 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 53 %

Touching and printing out SMI info

tt-smi reset was successful