Incorporate ttir_builder generated TTNNs as Load/Execute Tests in Explorer #2123
Conversation
vprajapati-tt
requested review from
odjuricicTT,
tt-mpantic,
sdjordjevicTT,
vcanicTT,
nsmithtt,
tapspatel and
vmilosevic
as code owners
|
I'm honestly stuck with this |
|
|
||
| def test_execute_and_check_accuracy_data_exists(): | ||
| # Get the test_mnist path | ||
| test_mnist_path = GET_TTNN_TEST() |
There was a problem hiding this comment.
Can we hardcode this path instead of searching for it?
There was a problem hiding this comment.
It is technically a build artifact, so I'm hesitant to hardcode the path. I think searching for it makes running tests locally easier as well.
| test_mnist_path, {"optimizationPolicy": "Optimizer Disabled"}, timeout=300 | ||
| ) | ||
| result = convert_command_and_assert(test_mnist_path) | ||
| if "accuracy_data" not in result["graphs"][0]["overlays"]: |
There was a problem hiding this comment.
Does the generated flatbuffer in CI have golden data? Does it have location data?
Are you able to check this from CI artifacts?
There was a problem hiding this comment.
I think it should have loc / golden data, I'll check the artifacts.
There was a problem hiding this comment.
CI Artifacts contain both location + Golden Data. Perf data is printing out correctly, it is very bizarre that accuracy is not. I've run the test countless times locally and I'm unable to reproduce this behaviour. I wonder if there is some cached explorer functionality that it is invoking instead of this...
There was a problem hiding this comment.
Ran the artifact locally through tt-explorer. accuracy_data was present.
There was a problem hiding this comment.
Does it make sense to try putting the flatbuffer in the github repo and running the test on that? Curious if that would solve the problem.
I wonder if there is some cached explorer functionality that it is invoking instead of this...
We restart the model explorer server on each test, so i would suspect that there is no caching in that case.
You could also try to comment out all other tests and run just this one to see if something changes.
ttir_builderinto Explorer CI. #2079