Generate and update models ops tests and reimplement the models ops test failure updation script #1234
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vmilosevic
force-pushed
the
generate_models_ops_test
branch
from
fbea0ab to
81d6bc2
Compare
vmilosevic
requested review from
nvukobratTT,
pilkicTT and
dgolubovicTT
as code owners
|
||||||||||||||
2 similar comments
|
||||||||||||||
|
||||||||||||||
chandrasekaranpradeep
force-pushed
the
generate_models_ops_test
branch
from
81d6bc2 to
9b8dd52
Compare
|
||||||||||||||
3 similar comments
|
||||||||||||||
|
||||||||||||||
|
||||||||||||||
chandrasekaranpradeep
force-pushed
the
generate_models_ops_test
branch
from
10d85ec to
02b119d
Compare
|
||||||||||||||
chandrasekaranpradeep
force-pushed
the
generate_models_ops_test
branch
2 times, most recently
from
d481e6e to
4a493d3
Compare
|
||||||||||||||
2 similar comments
|
||||||||||||||
|
||||||||||||||
chandrasekaranpradeep
force-pushed
the
generate_models_ops_test
branch
from
4a493d3 to
ea137a0
Compare
|
||||||||||||||
3 similar comments
|
||||||||||||||
|
||||||||||||||
|
||||||||||||||
|
||||||||||||||
3 similar comments
|
||||||||||||||
|
||||||||||||||
|
||||||||||||||
chandrasekaranpradeep
force-pushed
the
generate_models_ops_test
branch
2 times, most recently
from
f5af55f to
df1f1f5
Compare
|
||||||||||||||
|
||||||||||||||
|
||||||||||||||
nvukobratTT
reviewed
Feb 26, 2025
There was a problem hiding this comment.
Looks good!
General comment, let's see to create documentation for this one as well. Logic for generating model tests, collecting those data, etc. becomes more and more complex. It'll be good to have a robust documentation on this set of features.
Can you make a proposal how you would stracture doc? No need to have details for proposal. Let's agree on initial structure before we proceed further.
Note: Doc isn't major priority, let's finilize current efforts before we tackle with docs.
nvukobratTT
approved these changes
Feb 26, 2025
@nvukobratTT Created issue for documenting this ops test feature which also contains doc structure |
chandrasekaranpradeep
force-pushed
the
generate_models_ops_test
branch
from
612daa8 to
a1fcd29
Compare
|
||||||||||||||
1 similar comment
|
||||||||||||||
|
||||||||||||||
1 similar comment
|
||||||||||||||
… reshape ops test due to segmentation fault in CI
chandrasekaranpradeep
force-pushed
the
generate_models_ops_test
branch
from
a1fcd29 to
264b3a8
Compare
|
||||||||||||||
1 similar comment
|
||||||||||||||
|
||||||||||||||
1 similar comment
|
||||||||||||||
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The
scripts/model_analysis/models_ops_test_failure_update.pyscript automates the extraction of detailed test failure information from pytest logs by identifying test cases along with their statuses (FAILED, XFAIL, or SKIP) and corresponding error messages, then compiles this data into a comprehensive Excel report with columns for test cases, marker statuses, and reasons. This report enables users to manually refine error messages by updating the "Reason" column, and once revised, the script can be rerun using the--use_reportoption with thereport file pathto automatically update the model ops test files with marker and more descriptive failure messagesIndexError: index 563 is out of bounds for dimension 0 with size 448) in evaluation. The adv_index ops takes two inputs one is reference tensor and other is indicies tensor based upon the indicies tensor values will pick the reference tensor in respective dim but the indicies values exceeds the range of reference tensor. To resolve these issue, added pytest param max_int which will be used by the create_from_shape method in forge/forge/tensor.py to generate the indices tensor values with the range of the reference tensorProcess:
Triggered the nightly models ops pipeline for collecting the pytest logs
Pipeline: https://github.com/tenstorrent/tt-forge-fe/actions/runs/13536357567
Collected logs:
test-log-n150-1
test-log-n150-2
test-log-n150-3
test-log-n150-4
Using the collected pytest logs, generated excel report by using
scripts/model_analysis/models_ops_test_failure_update.pyscriptCommand:
python scripts/model_analysis/models_ops_test_failure_update.py --log_files ci_logs/pytest_1.log ci_logs/pytest_2.log ci_logs/pytest_3.log ci_logs/pytest_4.logGenerated models ops test report:
model_ops_tests_report.xlsx
Anlalyzed the excel report and update the models ops tests with xfail marker and failure reason by using the
scripts/model_analysis/models_ops_test_failure_update.pyscript withuse_reportoption enabledCommand:
python scripts/model_analysis/models_ops_test_failure_update.py --report_file_path model_ops_tests_report.xlsx --use_reportTriggered the nightly models ops pipeline and all the test are passed.
Pipeline: https://github.com/tenstorrent/tt-forge-fe/actions/runs/13540060061
Note:
This models ops test takes around 2hrs to run in push pipeline, so created a separate workflow to run this model ops test- #1316. So renamed the model ops test marker from
pushtonightly_models_ops