Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SGraph (re-)build optimisations #449

Merged
merged 4 commits into from
Dec 12, 2024
Merged

Conversation

reuterbal
Copy link
Collaborator

While profiling the building of the SGraph, I noticed two things:

  1. A non-negligible amount of time was spent on checking if items are part of any of the exclusion lists. The reason that this is costly is the pattern support, which incurs some overhead for repeatedly compiling search patterns. This can be partially reduced by using fnmatch.filter instead of a comprehension with any. On top of that, I tried reducing the amount of pattern compilations overall via some reordering or caching, however, since fnmatch already comes with a built-in pattern cache, this did worsen performance in all cases.
  2. The Module class includes module variables in the list of definitions, which is correct from a Fortran point-of-view but not required for the Scheduler, where we don't capture explicitly the dependency on individual variables but only the module itself. Hence, searching directly only for procedures, interfaces and typedefs brings the search effort down considerably.

Depending on the structure of the graph, this dropped graph building time by 10-20% for me. Much bigger gains could be achieved by caching definitions or dependencies, but this seems a little trickier to do.

@reuterbal reuterbal requested a review from mlange05 November 24, 2024 20:38
Copy link

Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/449/index.html

Copy link
Collaborator

@mlange05 mlange05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me and local ecphys regression also seemed fine. Small, but noticeable improvements during Loki processing - so very much appreciate this.

Will formally approve once test case green, but otherwise GTG from me.

@reuterbal reuterbal force-pushed the nabr-sgraph-rebuild-optimisation branch from dbea220 to aacccfc Compare December 10, 2024 16:39
Copy link

codecov bot commented Dec 10, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.29%. Comparing base (e9760e8) to head (aacccfc).
Report is 5 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #449   +/-   ##
=======================================
  Coverage   93.28%   93.29%           
=======================================
  Files         220      220           
  Lines       41224    41223    -1     
=======================================
  Hits        38457    38457           
+ Misses       2767     2766    -1     
Flag Coverage Δ
lint_rules 96.39% <ø> (ø)
loki 93.24% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@reuterbal
Copy link
Collaborator Author

Thanks! I fixed the failing test - turns out the new logic for yielding definitions for Module items removes some redundant reported items.

@reuterbal reuterbal requested a review from mlange05 December 10, 2024 20:18
Copy link
Collaborator

@mlange05 mlange05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic! Confirmed still functional and faster than before. All GTG. :shipit:

@mlange05 mlange05 added the ready to merge This PR has been approved and is ready to be merged label Dec 11, 2024
@reuterbal reuterbal merged commit fe6bf33 into main Dec 12, 2024
13 checks passed
@reuterbal reuterbal deleted the nabr-sgraph-rebuild-optimisation branch December 12, 2024 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready to merge This PR has been approved and is ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants