(Single-card) Tests for new models #230
Annotations
5 errors, 1 warning, and 6 notices
|
Complete runner
This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.
|
|
Complete runner
Process completed with exit code 1.
|
|
Run frequent reg tests scripts:
tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L46
test_run_resnet50_trace_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size
info:
Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region.
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5ceb) [0x7f7b863e9ceb]
--- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x1a0043) [0x7f7b862e4043]
--- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f7b891e1e07]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f7b891e1d0e]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f7b875e2d06]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f7bd0061a8b]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f7bd0060d10]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5136) [0x7f7bd00d0336]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f7bd006145a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7f7bd00cb5b3]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f7bd00ca5e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f7bd00614ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f7bd00620c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f7bd00cd570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f7bd00cab92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f7bd00614ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f7bd00620c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f7bd00cd570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f7bd00cab92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f7bd00614ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f7bd00cfcfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f7bd006145a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x1339c8) [0x7f7bd00639c8]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f7bd00cfcfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f7bd00ca5e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f7bd00614ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f7bd0060908]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(
|
|
Run frequent reg tests scripts:
tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L87
test_run_resnet50_trace_2cqs_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size
info:
Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region.
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5ceb) [0x7f4013963ceb]
--- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x1a0043) [0x7f401385e043]
--- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f401675be07]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f401675bd0e]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f4014b5cd06]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f405d5d8a8b]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f405d5d7d10]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5136) [0x7f405d647336]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f405d5d845a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7f405d6425b3]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f405d6415e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f405d5d84ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f405d5d90c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f405d644570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f405d641b92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f405d5d84ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f405d5d90c4]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2370) [0x7f405d644570]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f405d641b92]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f405d5d84ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f405d646cfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f405d5d845a]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x1339c8) [0x7f405d5da9c8]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4afc) [0x7f405d646cfc]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f405d6415e1]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f405d5d84ee]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f405d5d7908]
--- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so
|
|
Run frequent reg tests scripts
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
|
|
Run /./.github/actions/upload-artifact-with-job-uuid
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
|
|
Run /./.github/actions/retry-command
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Disk usage is 78 %
|
|
Set up runner
Touching and printing out SMI info
|
|
Set up runner
tt-smi reset was successful
|
|
Set up runner
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Hugepages is now setup.
|
Loading