-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SGraph (re-)build optimisations #449
Conversation
Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/449/index.html |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good to me and local ecphys regression also seemed fine. Small, but noticeable improvements during Loki processing - so very much appreciate this.
Will formally approve once test case green, but otherwise GTG from me.
dbea220
to
aacccfc
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #449 +/- ##
=======================================
Coverage 93.28% 93.29%
=======================================
Files 220 220
Lines 41224 41223 -1
=======================================
Hits 38457 38457
+ Misses 2767 2766 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Thanks! I fixed the failing test - turns out the new logic for yielding definitions for Module items removes some redundant reported items. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fantastic! Confirmed still functional and faster than before. All GTG.
While profiling the building of the SGraph, I noticed two things:
fnmatch.filter
instead of a comprehension withany
. On top of that, I tried reducing the amount of pattern compilations overall via some reordering or caching, however, sincefnmatch
already comes with a built-in pattern cache, this did worsen performance in all cases.Module
class includes module variables in the list of definitions, which is correct from a Fortran point-of-view but not required for the Scheduler, where we don't capture explicitly the dependency on individual variables but only the module itself. Hence, searching directly only for procedures, interfaces and typedefs brings the search effort down considerably.Depending on the structure of the graph, this dropped graph building time by 10-20% for me. Much bigger gains could be achieved by caching definitions or dependencies, but this seems a little trickier to do.