Add workflow for running models ops test generated by model analysis … #1316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chandrasekaranpradeep
requested review from
vmilosevic,
nvukobratTT,
pilkicTT and
dgolubovicTT
as code owners
vmilosevic
approved these changes
Feb 25, 2025
chandrasekaranpradeep
force-pushed
the
pchandrasekaran/models_ops_workflow
branch
from
656f30b to
57ed837
Compare
nvukobratTT
reviewed
Feb 25, 2025
Sure @vmilosevic will remove it |
chandrasekaranpradeep
force-pushed
the
pchandrasekaran/models_ops_workflow
branch
from
57ed837 to
3c8bcda
Compare
|
||||||||||||||
|
||||||||||||||
|
||||||||||||||
|
||||||||||||||
nvukobratTT
approved these changes
Feb 25, 2025
chandrasekaranpradeep
force-pushed
the
pchandrasekaran/models_ops_workflow
branch
from
3c8bcda to
a7ec886
Compare
chandrasekaranpradeep
force-pushed
the
pchandrasekaran/models_ops_workflow
branch
from
a7ec886 to
e6eebcd
Compare
|
||||||||||||||
|
||||||||||||||
1 similar comment
|
||||||||||||||
|
||||||||||||||
chandrasekaranpradeep
deleted the
pchandrasekaran/models_ops_workflow
branch
chandrasekaranpradeep
added a commit
that referenced
this pull request
Feb 27, 2025
…est failure updation script (#1234) ### Summary: 1. Generated models ops tests by extracting the unique ops configurations across all the pytorch models present inside the forge/test/models directory path. 2. Skipped avgpool1d, avgpool3d, conv2d, pad and reshape ops test cases due to segmentation fault in CI and transpose ops test cases due to Insufficient host DRAM (requires 30 GB of memory) 3. Reimplemented the model ops test failure updation script: The `scripts/model_analysis/models_ops_test_failure_update.py` script automates the extraction of detailed test **failure information** from pytest logs by identifying test cases along with their statuses (**FAILED, XFAIL, or SKIP**) and corresponding **error messages**, then compiles this data into a comprehensive **Excel report** with columns for **test cases**, **marker statuses**, and **reasons**. This report enables users to manually refine error messages by updating the "Reason" column, and once revised, the script can be rerun using the `--use_report` option with the `report file path` to automatically update the model ops test files with marker and more descriptive failure messages 4. The adv_index ops test are failing with indexerror out of bound error(i.e `IndexError: index 563 is out of bounds for dimension 0 with size 448`) in evaluation. The adv_index ops takes two inputs one is reference tensor and other is indicies tensor based upon the indicies tensor values will pick the reference tensor in respective dim but the indicies values exceeds the range of reference tensor. To resolve these issue, added pytest param max_int which will be used by the create_from_shape method in forge/forge/tensor.py to generate the indices tensor values with the range of the reference tensor ### Process: 1. Triggered the nightly models ops pipeline for collecting the pytest logs **Pipeline:** https://github.com/tenstorrent/tt-forge-fe/actions/runs/13536357567 **Collected logs:** [test-log-n150-1](https://github.com/tenstorrent/tt-forge-fe/actions/runs/13536357567/artifacts/2653622726) [test-log-n150-2](https://github.com/tenstorrent/tt-forge-fe/actions/runs/13536357567/artifacts/2653622477) [test-log-n150-3](https://github.com/tenstorrent/tt-forge-fe/actions/runs/13536357567/artifacts/2653619334) [test-log-n150-4](https://github.com/tenstorrent/tt-forge-fe/actions/runs/13536357567/artifacts/2653707424) 2. Using the collected pytest logs, generated excel report by using `scripts/model_analysis/models_ops_test_failure_update.py` script **Command:** `python scripts/model_analysis/models_ops_test_failure_update.py --log_files ci_logs/pytest_1.log ci_logs/pytest_2.log ci_logs/pytest_3.log ci_logs/pytest_4.log` **Generated models ops test report:** [model_ops_tests_report.xlsx](https://github.com/user-attachments/files/18982115/model_ops_tests_report.xlsx) 3. Anlalyzed the excel report and update the models ops tests with xfail marker and failure reason by using the `scripts/model_analysis/models_ops_test_failure_update.py` script with `use_report` option enabled **Command:** `python scripts/model_analysis/models_ops_test_failure_update.py --report_file_path model_ops_tests_report.xlsx --use_report` 4. Triggered the nightly models ops pipeline and all the test are passed. **Pipeline:** https://github.com/tenstorrent/tt-forge-fe/actions/runs/13540060061 ### Note: This models ops test takes around 2hrs to run in push pipeline, so created a separate workflow to run this model ops test- #1316. So renamed the model ops test marker from `push` to `nightly_models_ops` --------- Co-authored-by: chandrasekaranpradeep <[email protected]> Co-authored-by: chandrasekaranpradeep <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created workflow for running model ops test generated(#1234) by model analysis pipeline.