(Single-card) Tests for new models #238
Annotations
5 errors, 1 warning, and 6 notices
|
Complete runner
This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.
|
|
Complete runner
Process completed with exit code 1.
|
|
Run frequent reg tests scripts:
tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L46
test_run_resnet50_trace_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size
info:
Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region.
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7faaed40eb8b]
--- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7faaed308ee3]
--- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7faaf0210e07]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7faaf0210d0e]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7faaee611d06]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7fab32074b5b]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7fab32073df0]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x512c) [0x7fab320e359c]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fab3207452a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b2) [0x7fab320de822]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fab320dd851]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fab32075194]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7fab320e07d7]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fab320dde02]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fab32075194]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7fab320e07d7]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fab320dde02]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7fab320e2f0d]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fab3207452a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133a98) [0x7fab32076a98]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7fab320e2f0d]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fab320dd851]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fab320745be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7fab320739e8]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(
|
|
Run frequent reg tests scripts:
tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L87
test_run_resnet50_trace_2cqs_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size
info:
Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region.
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f3dc1618b8b]
--- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f3dc1512ee3]
--- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f3dc441ae07]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f3dc441ad0e]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f3dc281bd06]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f3e0627bb5b]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f3e0627adf0]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x512c) [0x7f3e062ea59c]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f3e0627b52a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b2) [0x7f3e062e5822]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f3e062e4851]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f3e0627c194]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f3e062e77d7]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f3e062e4e02]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f3e0627c194]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f3e062e77d7]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f3e062e4e02]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f3e062e9f0d]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f3e0627b52a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133a98) [0x7f3e0627da98]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f3e062e9f0d]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f3e062e4851]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f3e0627b5be]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f3e0627a9e8]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so
|
|
Run frequent reg tests scripts
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
|
|
Run /./.github/actions/upload-artifact-with-job-uuid
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
|
|
Run /./.github/actions/retry-command
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Disk usage is 69 %
|
|
Set up runner
Touching and printing out SMI info
|
|
Set up runner
tt-smi reset was successful
|
|
Set up runner
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Hugepages is now setup.
|
Loading