test_convnet_mnist.py in ttnn integration tests is starting to fail on all cards

[tt-rkim]

Not sure what's going on... there seems to be some interplay issues between tests.

For example, we see this often: https://github.com/tenstorrent/tt-metal/actions/runs/12782446765/job/35685828403

but stress tests are ok:

e150: https://github.com/tenstorrent/tt-metal/actions/runs/12796843809
n150: https://github.com/tenstorrent/tt-metal/actions/runs/12793625104

So not sure what's happening 🤷

Skipping for now

cc: @mywoodstock @pavlejosipovic

[tt-rkim]

First noticed it fail on run 6924 of nightly

but re-runs of 6923 and 6922 are no good either...

so it started more often around this time. Unsure what's happening. Is there a model cache for this?

[pavlejosipovic]

I can't run this model locally.
First one need to figure out how to get convnet_mnist.pt which isn't available on ird machines I'm working on (or any ird machine?).
Fortunately I have this file stashed locally that I got over slack from @vigneshkeerthivasanx.
Second thing is model tries to download mnist dataset from http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
And this is just failing for me.

[pavlejosipovic]

@tt-rkim let's disable this on WH and GS in every place, it's blocking device perf pipeline

[tt-rkim]

Oh device perf? OK one sec

[tt-rkim]

mnist is also causing issues despite the skip.

Refer to this re-run of stuff: https://github.com/tenstorrent/tt-metal/actions/runs/12777133171 - it was passing on attempt 1, then it all fails on attempt 2

Something whack is going on in the environment.

[tt-rkim]

I've skipped both mnist and convnet mnist

@eyonland @boris-drazic I believe either/both of you manage the third-party members who work on convnet mnist and mnist ttnn integration tests

Please triage and ask the writers to take a look at this and fix.

[tt-rkim]

I have skipped both tests.

[boris-drazic]

@kkeerthana0573 can you please take look? From schedules seems you are working on Convnet_MNIST

[tt-rkim]

Any update on this?

[kkeerthana0573]

@boris-drazic, @tt-rkim,
ConvMnist works as expected on N150, N300 machines locally.
However, I noticed that the model has been failing in Nightly CIs over the past few days. Interestingly, the ConvMnist tests were passing in the CIs yesterday -

• e2e-model-perf-single-card - GS, WH
• nightly-single-card - GS, N150, N300
• demos-single-card - N150, N300

I am retriggering the CIs to debug further.

[tt-rkim]

Please check out what's in main.

Also note that we only seem to see this behaviour if you run the whole ttnn integration test suite: tests/scripts/single_card/nightly/run_ttnn.sh

[kkeerthana0573]

On respective main,

• Last Week, the model passed here.
• Five days ago, it failed here.
• Yesterday, on the vignesh/ttnn_convnet_mnist_data_parallel branch, the model passed again here.

[tt-rkim]

Hmm... if you're able to run a couple more times to show it's not non-deterministic on a recent base of main, I will be more comfortable re-enabling the test

[kkeerthana0573]

I've triggered (Single-card) Nightly model and ttnn tests thrice and the model performed as expected in each run. Similarly, the model worked fine in the (Single-card) Device perf regressions. I've verified the model in (Single-card) Tests for new models too.

[tt-rkim]

I see... ok thank you

Please open up a PR for this so we can re-enable

And did you take any look at regular mnist?

[kkeerthana0573]

I've checked ConvMnist today. I will verify Mnist and will create a PR.

[Sudharsan-V]

cc: @mbahnasTT @saichandax

[tt-rkim]

Thank you, please keep us updated

[Sudharsan-V]

@tt-rkim ,
Both the Nightly and Device perf CIs are successfully passing for MNIST and ConvNet MNIST.

• Nightly CI Nightly CI and ttnn tests.
• Device Perf Mnist, convnet_mnist

Note: Device perf CI is currently unstable. Therefore, the links for mnist and convnet_mnist have been highlighted and shared shared above

Corresponding PR #16965