Extract Full Model Execution Tests into Nightly Tests

[jameszianxuTT]

Ticket

#378

Problem description

Full model execution tests are only attempted in push/pr triggered
workflows, but not in nightlies. This complicates queries and provides
no single source of truth on actual full model execution status from
tt-torch main.

What's changed

Duplicate the workflow in run-tests.yml into a self contained
job that can be run in nightly tests. Execution tests are refactored
to be run in parallel.

Checklist

• New/Existing tests provide coverage for changes

[jameszianxuTT]

Two Additional Notes

1. I have manually run the Nightly Tests workflow on a subset of the models to ensure this change doesn't break nightlies.
   See: https://github.com/tenstorrent/tt-torch/actions/runs/13575338428. This works.

2. I am disabling the torchvision execution test just for nightlies as those will add a significant amount of time to the nightly tests. Average test runtime is ~5m with the longest at ~10m and are run in parallel. The torchvision tests collectively take a bit more than an hour to run.

[jameszianxuTT]

A later PR should refactor the run-tests.yml to run those in parallel as well.

They should also probably be separate from the nightly tests since the number of models want to execute e2e should grow and not all of them need to be included in the on PR / on Push checks.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.81%. Comparing base (186ad28) to head (99e76fc).

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #379   +/-   ##
=======================================
  Coverage   75.81%   75.81%           
=======================================
  Files           8        8           
  Lines        1199     1199           
=======================================
  Hits          909      909           
  Misses        290      290           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Torch Tests	435 ran	428 passed	7 skipped	0 failed

Test	Result
No test annotations available

[github-actions]

	Tests	Passed ✅	Skipped ⚠️	Failed
TT-Torch Tests	435 ran	428 passed	7 skipped	0 failed

Test	Result
No test annotations available

[jameszianxuTT]

The most recent verification run is here: https://github.com/tenstorrent/tt-torch/actions/runs/13589631720

Here, I rerun nightly tests fully, and stop it when I'm sure everything has been correctly parsed and can run to avoid locking up CI.

The full execution test job I added does pass in this context.

[brataTT]

why are torchvision tests commented out?

[jameszianxuTT]

The torchvision tests take over an hour to run, and I don't want to extend nightly testing too much. All the other tests take <10m and are run in parallel.

I am not removing functionality from the onPR and onPush workflows, just duplicating a subset of their functionality to add more test coverage to the nightly tests. (Nightly tests do not currently have any full model execution tests).

The list of execution tests run in nightly will likely change anyways, and I don't know which ones are the most important. I've included most of the ones run in the onPR/onPush workflows as a proof-of-concept demonstration that this added CI job does work, with the assumption that this list of tests will likely be changed once Aleks gets back.

ref