(Single-card) Tests for new models #245
Annotations
5 errors, 1 warning, and 6 notices
|
Complete runner
This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.
|
|
Complete runner
Process completed with exit code 1.
|
|
Run frequent reg tests scripts:
tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L46
test_run_resnet50_trace_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size
info:
Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region.
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f51a5283b8b]
--- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f51a517dee3]
--- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f51a807ae07]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f51a807ad0e]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f51a647bd06]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f51eeef5a8b]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f51eeef4d10]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5136) [0x7f51eef64336]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f51eeef545a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7f51eef5f5b3]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f51eef5e5e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f51eeef54ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f51eeef60c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f51eef61570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f51eef5eb92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f51eeef54ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f51eeef60c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f51eef61570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f51eef5eb92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f51eeef54ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f51eef63cfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f51eeef545a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x1339c8) [0x7f51eeef79c8]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f51eef63cfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f51eef5e5e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f51eeef54ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f51eeef4908]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(
|
|
Run frequent reg tests scripts:
tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L87
test_run_resnet50_trace_2cqs_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size
info:
Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region.
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f1659a9ab8b]
--- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f1659994ee3]
--- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f165c89fe07]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f165c89fd0e]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f165aca0d06]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f16a3710a8b]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f16a370fd10]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5136) [0x7f16a377f336]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f16a371045a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7f16a377a5b3]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f16a37795e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f16a37104ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f16a37110c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f16a377c570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f16a3779b92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f16a37104ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f16a37110c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f16a377c570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f16a3779b92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f16a37104ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f16a377ecfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f16a371045a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x1339c8) [0x7f16a37129c8]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f16a377ecfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f16a37795e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f16a37104ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f16a370f908]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so
|
|
Run frequent reg tests scripts
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
|
|
Run /./.github/actions/upload-artifact-with-job-uuid
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
|
|
Run /./.github/actions/retry-command
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Disk usage is 83 %
|
|
Set up runner
Touching and printing out SMI info
|
|
Set up runner
tt-smi reset was successful
|
|
Set up runner
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Hugepages is now setup.
|
Loading