Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TMVA][SOFIE] Memory Optimization for Intermediate tensors - v1 #17532

Open
wants to merge 36 commits into
base: master
Choose a base branch
from

Conversation

sanjibansg
Copy link
Contributor

This PR brings the first version for the memory optimization for intermediate tensors.

Currently, in the inference code, all the intermediate tensors are initialized at first and are allocated memory, thus no memory reuse takes place. Since not all the intermediate tensors are required till the end, we can flush some of them, and reuse the memory that they release.

This first edition of memory optimization for intermediate tensors uses a simple mechanism, where the total memory required is first calculated along with the count of operators who need a particular intermediate tensor as their input.

While adding a operator into the RModel during the Parsing stage, its input operators are taken into account into an unordered map (added as a new key-value pair if not already present, otherwise frequency is incremented). During the initialize phase, we calculate the total memory required by the tensors. We keep two separate containers - total memory (for calculating the total memory required by all the tensors) and available memory (for accounting all the reusable memory - memory that was flushed and can be reused). During the generate phase, we Initialize the intermediate tensors when a particular operator needs it as an output, and flush them when no operator no longer needs it as an input.

@sanjibansg sanjibansg changed the title [TMVA][SOFIE] Memory Optimization for Intermediate tensors - v1 [WIP] [TMVA][SOFIE] Memory Optimization for Intermediate tensors - v1 Jan 27, 2025
Copy link

github-actions bot commented Jan 27, 2025

Test Results

    18 files      18 suites   4d 2h 36m 12s ⏱️
 2 722 tests  2 721 ✅ 0 💤 1 ❌
47 298 runs  47 292 ✅ 0 💤 6 ❌

For more details on these failures, see this check.

Results for commit 310e38b.

♻️ This comment has been updated with latest results.

@sanjibansg sanjibansg changed the title [WIP] [TMVA][SOFIE] Memory Optimization for Intermediate tensors - v1 [TMVA][SOFIE] Memory Optimization for Intermediate tensors - v1 Feb 3, 2025
@sanjibansg sanjibansg marked this pull request as ready for review February 3, 2025 12:02
@sanjibansg sanjibansg requested a review from lmoneta as a code owner February 3, 2025 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants