Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Enable resize scheduler by default #3848

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

naoyam
Copy link
Collaborator

@naoyam naoyam commented Feb 7, 2025

No description provided.

@naoyam
Copy link
Collaborator Author

naoyam commented Feb 7, 2025

!test

Copy link

github-actions bot commented Feb 7, 2025

Review updated until commit c79e234

Description

  • Enable resize scheduler by default

  • Update option handling for resize scheduler

  • Modify logic in preseg_passes and scheduler

  • Adjust tests to reflect new default


Changes walkthrough 📝

Relevant files
Configuration changes
3 files
options.cpp
Move resize scheduler option to disable options                   
+1/-1     
options.h
Move resize scheduler option to disable options                   
+1/-1     
loop_domain_scheduler.h
Update cancelReshapeInLoopDomains signature                           
+6/-1     
Enhancement
4 files
pre_segmenter.cpp
Update resize scheduler check                                                       
+1/-1     
resize.cpp
Update resize scheduler logic                                                       
+8/-6     
loop_domain_scheduler.cpp
Enhance cancelReshapeInLoopDomains with skip_innermost_id
+25/-4   
resize_utils.cpp
Update scheduleLoopDomainsBy call                                               
+2/-1     
Tests
3 files
test_move_pad.cpp
Disable resize scheduler in MovePadTest                                   
+7/-1     
test_resize.cpp
Update tests to disable resize scheduler                                 
+6/-18   
test_rope.cpp
Update RopeTest to use ResizeTest                                               
+1/-9     

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests
⚡ Recommended focus areas for review

Default Behavior

The PR enables the resize scheduler by default, but it seems that the scheduler is still being disabled in several test cases. Ensure that the default behavior is correctly implemented and that tests are updated to reflect this change.

if (isOptionDisabled(DisableOption::ResizeScheduler)) {
  scheduler_debug_utils::canScheduleRejectReason(schedulerType(), "Disabled");
  return false;
}
Innermost ID Handling

The cancelReshapeInLoopDomains function now includes a skip_innermost_id parameter. Ensure that this change does not introduce any unintended behavior and that the logic is correctly implemented.

}

// Update the loop domain by each of the reshape exprs in a
// reverse topological order.
auto reshape_exprs = DependencyCheck::getAllExprsBetween(
    {reshape_out->getRootDomain().begin(),
     reshape_out->getRootDomain().end()},
    {reshape_out->getLogicalDomain().begin(),
     reshape_out->getLogicalDomain().end()});

std::unordered_set<Expr*> reshape_exprs_with_innermost_logical_id_set;
if (skip_innermost_id) {
  auto reshape_exprs_with_innermost_logical_id =
      DependencyCheck::getAllExprsBetween(
          {reshape_out->getRootDomain().begin(),
           reshape_out->getRootDomain().end()},
          {reshape_out->getLogicalDomain().back()});
  reshape_exprs_with_innermost_logical_id_set = {
      reshape_exprs_with_innermost_logical_id.begin(),
      reshape_exprs_with_innermost_logical_id.end()};
}

auto reshape_out_loop_domain = reshape_out->getLoopDomain();

for (auto reshape_exprs_it = reshape_exprs.rbegin();
     reshape_exprs_it != reshape_exprs.rend();
     ++reshape_exprs_it) {
  auto reshape_expr = *reshape_exprs_it;

  if (skip_innermost_id &&
      reshape_exprs_with_innermost_logical_id_set.count(reshape_expr)) {
    continue;
  }

  // If any of the output IDs of reshape_expr is not found in
  // cancellable_ids, that means the expr cannot be cancelled.
  if (std::any_of(
Test Setup

The ResizeTest and ResizeSchedulerTest classes now disable the resize scheduler by default. Verify that these tests are still valid and that disabling the scheduler does not affect the test outcomes.

using ResizeTest = NVFuserTest;

using ResizeSchedulerTest = NVFuserFixtureParamTest<bool>;

Base automatically changed from fix_rope_llama3_bwd to main February 7, 2025 17:05
@naoyam naoyam force-pushed the enable_resize_scheduler_by_default branch from 006ef2b to ae17a5a Compare February 7, 2025 17:08
@naoyam
Copy link
Collaborator Author

naoyam commented Feb 10, 2025

!test

1 similar comment
@naoyam
Copy link
Collaborator Author

naoyam commented Feb 10, 2025

!test

@naoyam naoyam force-pushed the enable_resize_scheduler_by_default branch 2 times, most recently from 9044331 to dd5d144 Compare February 12, 2025 21:42
@naoyam naoyam force-pushed the enable_resize_scheduler_by_default branch from dd5d144 to 4cb7fbd Compare February 12, 2025 21:43
@naoyam
Copy link
Collaborator Author

naoyam commented Feb 13, 2025

!test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant