test_all_gather_on_n300_post_commit[silicon_arch_name=wormhole_b0-silicon_arch_wormhole_b0=True-enable_async=True-num_iters=1-mem_config=MemoryConfig(memory_layout=TensorMemoryLayout::INTERLEAVED,buffer_type=BufferType::L1,shard_spec=std::nullopt)-input_dtype=DataType.BFLOAT8_B-num_devices=2-num_links=1-output_shape=[1, 1, 32, 704]-dim=3-layout=Layout.TILE] RuntimeError: TT_FATAL @ /work/ttnn/cpp/ttnn/operations/ccl/all_gather/device/all_gather_op.cpp:242: num_devices > 1 info: all_gather op will only work for num_devices > 1, but has 1 backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xbf32f0) [0x7fb175a472f0] --- ttnn::operations::ccl::all_gather(tt::tt_metal::Tensor const&, int, unsigned int, std::__1::optional<tt::tt_metal::MemoryConfig> const&, std::__1::optional<unsigned long>, std::__1::optional<unsigned long>, ttnn::ccl::Topology) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x232ea86) [0x7fb177182a86] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x232e927) [0x7fb177182927] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x232e5f5) [0x7fb1771825f5] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0x232e50e) [0x7fb17718250e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/.local/lib/python3.8/site-packages/ttnn/_ttnn.cpython-38-x86_64-linux-gnu.so(+0xc1827e) [0x7fb175a6c27e] --- /usr/bin/python3(PyCFunction_Call+0x59) [0x5f59b9] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3() [0x50adc8] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3() [0x59c85c] --- /usr/bin/python3(PyObject_Call+0x27e) [0x5f534e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3() [0x59c54f] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x62f5) [0x570d65] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1882) [0x56c2f2] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1882) [0x56c2f2] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(PyObject_Call+0x1f7) [0x5f52c7] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x1f35) [0x56c9a5] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3() [0x50a59e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x5809) [0x570279] --- /usr/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x56946a] --- /usr/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5f43] --- /usr/bin/python3() [0x59c54f] --- /usr/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f658e] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x62f5) [0x570d65] --- /usr/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5d66] --- /usr/bin/python3(_PyEval_EvalFrameDefault+0x859) [0x56b2c9] --- /usr/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5d66] --- /usr/bin/python3

[DEPRECATION] This action is deprecated. Please migrate to reading the Docker image from the pipeline.

Disk usage is 65 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available