(Single-card) Model perf tests #7760
Annotations
4 errors, 10 warnings, and 6 notices
|
Check perf report exists
Process completed with exit code 2.
|
|
Run performance regressions:
models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L86
test_perf_2cqs[32-0.006-30-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/allocator/bank_manager.cpp:114: num_shards.value() <= num_compute_banks
info:
Expected number of shards 130 to be less than or equal to total number of L1 banks 64 in compute cores
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xe977c9) [0x7fa322e577c9]
--- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::__1::optional<unsigned int>)
--- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*)
--- tt::tt_metal::v0::Buffer::allocate_impl()
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x1befc0) [0x7fa321c5cfc0]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x1a59c1) [0x7fa321c439c1]
--- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::__1::optional<tt::tt_metal::ShardSpecBuffer> const&, std::__1::optional<bool>, std::__1::optional<tt::tt_metal::SubDeviceId>)
--- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&)
--- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*)
--- ttnn::operations::data_movement::ReshardDeviceOperation::create_output_tensors(std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&) const
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xebc009) [0x7fa322e7c009]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x3a) [0x7fa324f3f15a]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2f7f09a) [0x7fa324f3f09a]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7fa324f3ea57]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2f7e40b) [0x7fa324f3e40b]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2f7e0eb) [0x7fa324f3e0eb]
--- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xebb7ac) [0x7fa322e7b7ac]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xeba17b) [0x7fa322e7a17b]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2a9eef8) [0x7fa324a5eef8]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2a9ecb5) [0x7fa324a5ecb5]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2a9ec1d) [0x
|
|
Run performance regressions:
models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L114
test_perf_trace_2cqs[32-0.006-30-device_params0]
RuntimeError: TT_FATAL @ /work/tt_metal/impl/allocator/bank_manager.cpp:114: num_shards.value() <= num_compute_banks
info:
Expected number of shards 130 to be less than or equal to total number of L1 banks 64 in compute cores
backtrace:
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xe977c9) [0x7fa322e577c9]
--- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::__1::optional<unsigned int>)
--- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*)
--- tt::tt_metal::v0::Buffer::allocate_impl()
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x1befc0) [0x7fa321c5cfc0]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x1a59c1) [0x7fa321c439c1]
--- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::__1::optional<tt::tt_metal::ShardSpecBuffer> const&, std::__1::optional<bool>, std::__1::optional<tt::tt_metal::SubDeviceId>)
--- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&)
--- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*)
--- ttnn::operations::data_movement::ReshardDeviceOperation::create_output_tensors(std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&) const
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xebc009) [0x7fa322e7c009]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x3a) [0x7fa324f3f15a]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2f7f09a) [0x7fa324f3f09a]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(_ZN4ttnn16device_operation6detail6invokeIN2tt8tt_metal9operation23OldInfraDeviceOperationINSt3__16vectorINS4_6TensorENS7_9allocatorIS9_EEEEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSE_22operation_attributes_tERKNSE_13tensor_args_tE+0x197) [0x7fa324f3ea57]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2f7e40b) [0x7fa324f3e40b]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2f7e0eb) [0x7fa324f3e0eb]
--- std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> tt::tt_metal::operation::run<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>(tt::tt_metal::operation::DeviceOperation<std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>>>&&, std::__1::vector<tt::tt_metal::Tensor, std::__1::allocator<tt::tt_metal::Tensor>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor const>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor const>>> const&, std::__1::vector<std::__1::optional<tt::tt_metal::Tensor>, std::__1::allocator<std::__1::optional<tt::tt_metal::Tensor>>> const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>)
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xebb7ac) [0x7fa322e7b7ac]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0xeba17b) [0x7fa322e7a17b]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2a9eef8) [0x7fa324a5eef8]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2a9ecb5) [0x7fa324a5ecb5]
--- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/ttnn/ttnn/_ttnn.so(+0x2a9ec1
|
|
Run performance regressions
Process completed with exit code 1.
|
|
Run performance regressions:
models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L11
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
|
|
Run performance regressions:
python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
|
|
Run performance regressions:
models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L38
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
|
|
Run performance regressions:
python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
|
|
Run performance regressions:
models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L38
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
|
|
Run performance regressions:
python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
|
|
Run performance regressions:
models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L71
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
|
|
Run performance regressions:
python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
|
|
Run performance regressions:
models/demos/wormhole/resnet50/tests/test_perf_e2e_resnet50.py#L71
record_property is incompatible with junit_family 'xunit2' (use 'legacy' or 'xunit1')
|
|
Run performance regressions:
python_env/lib/python3.8/site-packages/huggingface_hub/file_download.py#L1142
`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
|
|
Run /./.github/actions/ensure-active-weka-mount
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Disk usage is 62 %
|
|
Set up runner
Touching and printing out SMI info
|
|
Set up runner
tt-smi reset was successful
|
|
Set up runner
Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available
|
|
Set up runner
Hugepages is now setup.
|
Loading