[DRAFT] [WIP] Path exploration for TT-NN x TT-Mesh Integration #18067
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -614,7 +606,7 @@ Tensor to_host_mesh_tensor(const Tensor& tensor, bool blocking) { | |||
|
|
|||
| mesh_cq.enqueue_read_shards(shard_data_transfers, mesh_buffer, /*blocking=*/true); | |||
|
|
|||
| MultiDeviceHostStorage host_storage(storage.strategy, std::move(buffers), std::move(specs)); | |||
| MultiDeviceHostStorage host_storage(AllGatherTensor{}, std::move(buffers), std::move(specs)); | |||
There was a problem hiding this comment.
Fyi, on another branch I am attempting to get rid of "strategy". I think we only need a shape to track which devices the tensor was uploaded. It can be a full mesh shape or a submesh.
There was a problem hiding this comment.
that would be great!
| std::vector<T> host_buffer; | ||
| const auto& shard_tensor_spec = storage.specs.at(id); | ||
| const auto& shard_tensor_spec = tensor.get_tensor_spec(); |
There was a problem hiding this comment.
We will need the per-shard information, won't we? What is your plan here?
There was a problem hiding this comment.
pragmatically for now. I'll add support for it if it's really needed. For now, let's see how far I can get.
cfjchu
force-pushed
the
jchu/ttnn-integration-with-mesh
branch
4 times, most recently
from
7441d1b to
560ad32
Compare
| @@ -108,6 +80,7 @@ class Tensor { | |||
| explicit Tensor( | |||
| uint32_t num_buffers, std::optional<DistributedTensorConfig> distributed_tensor_config = std::nullopt); | |||
| explicit Tensor(const std::vector<IDevice*>& workers); | |||
cfjchu
force-pushed
the
jchu/ttnn-integration-with-mesh
branch
from
71a3899 to
bae493e
Compare
This reverts commit bae493e.
### Ticket #18360 ### Problem description Recently we disabled async mode for single device, by ignoring enable_async call for it, assuming multi-device customers make a call to MeshDevice enable_async. However in some places including our test we actually iterate over each individual device in the mesh and call enable_async on it, which is being ignored ### What's changed Make a single call to MeshDevice::enable_async instead of iterating over individual devices and calling Device::enable_async for each one of them ### Checklist - [ ] [All post commit CI passes](https://github.com/tenstorrent/tt-metal/actions/runs/13553947437) - [x] [T3K demo tests CI passes](https://github.com/tenstorrent/tt-metal/actions/runs/13553950838) - [x] New/Existing tests provide coverage for changes (cherry picked from commit 69a36b8)
lots of fun TMP
…-with-mesh-rebase
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
Link to Github Issue
Problem description
Provide context for the problem.
What's changed
Checklist