Skip to content

(Single-card) Tests for new models #237

(Single-card) Tests for new models

(Single-card) Tests for new models #237

Re-run triggered February 24, 2025 16:50
Status Failure
Total duration 3h 48m 23s
Artifacts 27

full-new-models-suite.yaml

on: workflow_dispatch
build-artifact-profiler  /  ...  /  check-docker-images
8s
build-artifact-profiler / build-docker-image / check-docker-images
build-artifact  /  ...  /  check-docker-images
8s
build-artifact / build-docker-image / check-docker-images
build-artifact-profiler  /  ...  /  🐳️ Build image
0s
build-artifact-profiler / build-docker-image / 🐳️ Build image
build-artifact  /  ...  /  🐳️ Build image
0s
build-artifact / build-docker-image / 🐳️ Build image
build-artifact-profiler  /  🛠️ Build Release ubuntu 20.04
3m 13s
build-artifact-profiler / 🛠️ Build Release ubuntu 20.04
build-artifact  /  🛠️ Build Release ubuntu 20.04
3m 1s
build-artifact / 🛠️ Build Release ubuntu 20.04
Matrix: device-perf-single-card / device-perf
Matrix: demos-single-card / single-card-demo-tests
Matrix: e2e-model-perf-single-card / models-perf
Matrix: nightly-single-card / fd-nightly
Matrix: nightly-single-card / nightly-wh-models
Matrix: nightly-single-card / nightly-wh-unstable-models
Fit to window
Zoom out
Zoom in

Annotations

27 errors, 136 warnings, and 174 notices
e2e-model-perf-single-card / cnn_javelin N300 WH B0
Process completed with exit code 2.
e2e-model-perf-single-card / cnn_javelin N300 WH B0: models/experimental/functional_unet/tests/test_unet_perf.py#L119
test_unet_trace_perf_multi_device[wormhole_b0-True-1-2-128-25.0-1650.0-False-device_params0] RuntimeError: bad_variant_access
e2e-model-perf-single-card / cnn_javelin N300 WH B0
Process completed with exit code 1.
device-perf-single-card / N300 WH B0 device perf
Process completed with exit code 2.
device-perf-single-card / N300 WH B0 device perf
Unable to download artifact(s): Artifact not found for name: eager-dist-ubuntu-20.04-any-profiler Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact. For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md
nightly-single-card / Nightly N300 functional_unet: tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py#L563
test_unet_trace_2cq_same_io_multi_device[wormhole_b0-True-1-2-128-device_params0-False] RuntimeError: bad_variant_access
nightly-single-card / Nightly N300 functional_unet: tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_upblock.py#L71
test_unet_upblock[device_params0-upblock2-32-132-20-32-1-2] AssertionError: 0.9989989668297883
nightly-single-card / Nightly N300 functional_unet
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
pcie-cards-are-being-used-cleanup
This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.
nightly-single-card / Nightly N300 functional_unet
Process completed with exit code 1.
pcie-cards-are-being-used-cleanup
This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.
nightly-single-card / Nightly N300 resnet50
Process completed with exit code 1.
nightly-single-card / Nightly N300 resnet50: tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L46
test_run_resnet50_trace_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f818b1b6b8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f818b0b0ee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f818dfb8e07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f818dfb8d0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f818c3b9d06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f81cfe1ab5b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f81cfe19df0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x512c) [0x7f81cfe8959c] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f81cfe1a52a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b2) [0x7f81cfe84822] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f81cfe83851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f81cfe1a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f81cfe1b194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f81cfe867d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f81cfe83e02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f81cfe1a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f81cfe1b194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f81cfe867d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f81cfe83e02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f81cfe1a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f81cfe88f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f81cfe1a52a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133a98) [0x7f81cfe1ca98] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f81cfe88f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f81cfe83851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f81cfe1a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f81cfe199e8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(
nightly-single-card / Nightly N300 resnet50: tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L87
test_run_resnet50_trace_2cqs_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f1546138b8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f1546032ee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f1548f3ae07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f1548f3ad0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f154733bd06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f158ad9ab5b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f158ad99df0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x512c) [0x7f158ae0959c] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f158ad9a52a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b2) [0x7f158ae04822] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f158ae03851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f158ad9a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f158ad9b194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f158ae067d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f158ae03e02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f158ad9a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f158ad9b194] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2367) [0x7f158ae067d7] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f158ae03e02] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f158ad9a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f158ae08f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f158ad9a52a] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133a98) [0x7f158ad9ca98] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4a9d) [0x7f158ae08f0d] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f158ae03851] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f158ad9a5be] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f158ad999e8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so
nightly-single-card / Nightly N300 resnet50
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
pcie-cards-are-being-used-cleanup
This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.
nightly-single-card / Nightly N150 functional_unet
Process completed with exit code 1.
nightly-single-card / Nightly N150 functional_unet: tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py#L476
test_unet_trace_2cq_same_io[1-2-128-device_params0] AssertionError: 0.8929509806743586
nightly-single-card / Nightly N150 functional_unet: tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_upblock.py#L71
test_unet_upblock[device_params0-upblock4-16-528-80-16-1-2] AssertionError: 0.998989618643751
nightly-single-card / Nightly N150 functional_unet
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
pcie-cards-are-being-used-cleanup
This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.
nightly-single-card / Nightly N150 resnet50
Process completed with exit code 1.
nightly-single-card / Nightly N150 resnet50: tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L46
test_run_resnet50_trace_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7f7670323b8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7f767021dee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7f767311ae07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7f767311ad0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7f767151bd06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7f76b9f9292b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7f76b9f91bc0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5130) [0x7f76ba001670] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f76b9f922fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7f76b9ffc8f3] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f76b9ffb921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f76b9f9238e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f76b9f92f64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7f76b9ffe8a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f76b9ffbed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f76b9f9238e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7f76b9f92f64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7f76b9ffe8a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7f76b9ffbed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f76b9f9238e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7f76ba001034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7f76b9f922fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133868) [0x7f76b9f94868] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7f76ba001034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7f76b9ffb921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7f76b9f9238e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7f76b9f917b8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(
nightly-single-card / Nightly N150 resnet50: tests/nightly/single_card/resnet50/models/demos/wormhole/resnet50/tests/test_resnet50_performant.py#L87
test_run_resnet50_trace_2cqs_inference[False-16-act_dtype0-weight_dtype0-math_fidelity0-device_params0] RuntimeError: TT_FATAL @ /work/tt_metal/impl/trace/trace.cpp:46: cq.device()->get_trace_buffers_size() <= trace_region_size info: Creating trace buffers of size 802816B on device 0, but only 800768B is allocated for trace region. backtrace: --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2a5b8b) [0x7fcfcb332b8b] --- tt::tt_metal::v0::Trace::initialize_buffer(tt::tt_metal::CommandQueue&, std::__1::shared_ptr<tt::tt_metal::TraceBuffer> const&) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x19fee3) [0x7fcfcb22cee3] --- tt::tt_metal::v0::Device::end_trace(unsigned char, unsigned int) --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adae07) [0x7fcfce129e07] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2adad0e) [0x7fcfce129d0e] --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xedbd06) [0x7fcfcc52ad06] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyCFunction_Call+0x6b) [0x7fd014fa992b] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_MakeTpCall+0x170) [0x7fd014fa8bc0] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x5130) [0x7fd015018670] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fd014fa92fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x3b3) [0x7fd0150138f3] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fd015012921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fd014fa938e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fd014fa9f64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7fd0150158a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fd015012ed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fd014fa938e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(PyVectorcall_Call+0x74) [0x7fd014fa9f64] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x2368) [0x7fd0150158a8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x8b2) [0x7fd015012ed2] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fd014fa938e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7fd015018034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xfa) [0x7fd014fa92fa] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(+0x133868) [0x7fd014fab868] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x4af4) [0x7fd015018034] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x301) [0x7fd015012921] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyFunction_Vectorcall+0x18e) [0x7fd014fa938e] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so.1.0(_PyObject_FastCallDict+0x238) [0x7fd014fa87b8] --- /home/ubuntu/actions-runner/_work/_tool/Python/3.8.18/x64/lib/libpython3.8.so
nightly-single-card / Nightly N150 resnet50
The action 'Run frequent reg tests scripts' has timed out after 30 minutes.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: tests/nightly/single_card/stable_diffusion/test_resnet_block_2d.py#L72
test_resnet_block_2d_512x512[2-2560-8-8-0-1-up-1280-device_params0] huggingface_hub.errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/models/CompVis/stable-diffusion-v1-4 <!DOCTYPE html> <html class="" lang="en"> <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no" /> <meta name="description" content="We're on a journey to advance and democratize artificial intelligence through open source and open science." /> <meta property="fb:app_id" content="1321688464574422" /> <meta name="twitter:card" content="summary_large_image" /> <meta name="twitter:site" content="@huggingface" /> <meta property="og:title" content="Hugging Face - The AI community building the future." /> <meta property="og:type" content="website" /> <title>Hugging Face - The AI community building the future.</title> <style> body { margin: 0; } main { background-color: white; min-height: 100vh; padding: 7rem 1rem 8rem 1rem; text-align: center; font-family: Source Sans Pro, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Arial, Noto Sans, sans-serif, Apple Color Emoji, Segoe UI Emoji, Segoe UI Symbol, Noto Color Emoji; } img { width: 6rem; height: 6rem; margin: 0 auto 1rem; } h1 { font-size: 3.75rem; line-height: 1; color: rgba(31, 41, 55, 1); font-weight: 700; box-sizing: border-box; margin: 0 auto; } p, a { color: rgba(107, 114, 128, 1); font-size: 1.125rem; line-height: 1.75rem; max-width: 28rem; box-sizing: border-box; margin: 0 auto; } .dark main { background-color: rgb(11, 15, 25); } .dark h1 { color: rgb(209, 213, 219); } .dark p, .dark a { color: rgb(156, 163, 175); } </style> <script> // On page load or when changing themes, best to add inline in `head` to avoid FOUC const key = "_tb_global_settings"; let theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "dark" : "light"; try { const storageTheme = JSON.parse(window.localStorage.getItem(key)).theme; if (storageTheme) { theme = storageTheme === "dark" ? "dark" : "light"; } } catch (e) {} if (theme === "dark") { document.documentElement.classList.add("dark"); } else { document.documentElement.classList.remove("dark"); } </script> </head> <body> <main> <img src="https://cdn-media.huggingface.co/assets/huggingface_logo.svg" alt="" /> <div> <h1>504</h1> <p>Gateway Timeout</p> </div> </main> </body> </html>
nightly-single-card / [Unstable] Nightly N150 stable_diffusion
Process completed with exit code 1.
e2e-model-perf-single-card / llm_javelin N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / llm_javelin N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / llm_javelin N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / llm_javelin N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / llm_javelin N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / llm_javelin N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / other N300 WH B0: models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L11
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
e2e-model-perf-single-card / other N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / other N300 WH B0: models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L38
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
e2e-model-perf-single-card / other N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / other N300 WH B0: models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L38
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
e2e-model-perf-single-card / other N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / other N300 WH B0: models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L71
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
e2e-model-perf-single-card / other N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
e2e-model-perf-single-card / other N300 WH B0: models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L98
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
e2e-model-perf-single-card / other N300 WH B0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N300 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_functionality: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L89
To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L100
To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N300_performance: models/demos/wormhole/mistral7b/demo/demo_with_prefill.py#L548
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
demos-single-card / N150: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / [Unstable] Nightly N150 mamba 5: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 mamba 5: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
demos-single-card / N150: python_env/lib/python3.8/site-packages/transformers/generation/utils.py#L4811
`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.
nightly-single-card / FD WH N150 ttnn nightly wormhole_b0: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 mamba 5: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 mamba 5: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: models/demos/wormhole/stable_diffusion/custom_preprocessing.py#L32
The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3637.)
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N300 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_embedding.py#L18
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_rms_norm.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_mlp.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_attention.py#L23
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_attention_prefill.py#L24
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_decoder.py#L23
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_decoder_prefill.py#L24
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_rms_norm.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_embedding.py#L18
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_mlp.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_rms_norm.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N150 llama3.2-1B: models/demos/llama3/tests/test_llama_attention.py#L23
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_mlp.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_attention.py#L23
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_attention_prefill.py#L24
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_decoder.py#L23
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_decoder_prefill.py#L24
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_rms_norm.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_mlp.py#L19
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 llama3.2-1B: models/demos/llama3/tests/test_llama_attention.py#L23
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 common_models: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / Nightly N300 functional_unet
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
nightly-single-card / Nightly N300 functional_unet: python_env/lib/python3.8/site-packages/_pytest/python.py#L199
Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.6406722068786621, inference_time=0.002381281927227974), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?
nightly-single-card / Nightly N300 functional_unet: python_env/lib/python3.8/site-packages/_pytest/python.py#L199
Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.6406722068786621, inference_time=0.002381281927227974), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?
nightly-single-card / Nightly N300 resnet50
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
nightly-single-card / Nightly N150 functional_unet
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
nightly-single-card / Nightly N150 resnet50
No files were found with the provided path: generated/test_reports/. No artifacts will be uploaded.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: models/demos/wormhole/stable_diffusion/custom_preprocessing.py#L32
The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3637.)
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
nightly-single-card / [Unstable] Nightly N150 stable_diffusion: python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 74 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 74 %
printing-smi-info-startup
Touching and printing out SMI info
attempting-reset-startup
Attempting to reset card(s). Sleeping first
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 82 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 53 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 83 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 58 %
disk-usage-after-startup
Disk usage is 62 %
hugepages-setup-success-startup
Hugepages is now setup.
printing-smi-info-startup
Touching and printing out SMI info
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
reset-successful-startup
tt-smi reset was successful
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
hugepages-setup-success-startup
Hugepages is now setup.
disk-usage-after-startup
Disk usage is 77 %
printing-smi-info-startup
Touching and printing out SMI info
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 70 %
printing-smi-info-startup
Touching and printing out SMI info
disk-usage-after-startup
Disk usage is 64 %
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-smi-info-startup
Touching and printing out SMI info
hugepages-setup-success-startup
Hugepages is now setup.
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
disk-usage-after-startup
Disk usage is 50 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 76 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 80 %
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
printing-smi-info-startup
Touching and printing out SMI info
disk-usage-after-startup
Disk usage is 60 %
reset-successful-startup
tt-smi reset was successful
printing-smi-info-startup
Touching and printing out SMI info
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
reset-successful-startup
tt-smi reset was successful
hugepages-setup-success-startup
Hugepages is now setup.
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 69 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
disk-usage-after-startup
Disk usage is 78 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 59 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 51 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 64 %
hugepages-setup-success-startup
Hugepages is now setup.
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 58 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 81 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 64 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 74 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
disk-usage-after-startup
Disk usage is 74 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 75 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 73 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 73 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 75 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.
weka-mount-hugepages-service-found
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
disk-usage-after-startup
Disk usage is 72 %
printing-smi-info-startup
Touching and printing out SMI info
reset-successful-startup
tt-smi reset was successful
hugepages-service-found-startup
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
hugepages-setup-success-startup
Hugepages is now setup.

Artifacts

Produced during runtime
Name Size
test_reports_24daf3a8-879b-4671-ac28-119ac4443287
1.32 KB
test_reports_336b371e-8d0d-4ab2-831f-a7dc3070cde0
8.76 KB