flux-top: command that gives insight into the current utilization of job resources

[SteVwonder]

As the title says, it would be cool to be able to do something like flux top $JOBID and see the aggregate cpu/gpu/mem usage of all the nodes in a job. Maybe flux-exec could be leveraged here to spawn a "daemon" on each node in a user's job. Those daemons could then collect utilization statistics, forward them up the overlay, where they are reduced and fed to the flux-top front-end. Alternatively, maybe we have an always loaded module that collects these stats, and the flux-top front-end would just filter to just the stats that apply the the user's particular job.

Opening because I just had a workflow user request this while debugging node OOMs. The traditional method of debugging an OOM (attach a parallel debugger) is hard since there are tons of apps/components running in this composite workflow. So just getting a feel for which of the components is using the most memory would be helpful.

[chu11]

I know flux-top has been discussed a few times, but never really put down the ideas fully on paper yet.

initial brainstorming idea for the short term. Could flux-exec be used to run some random utilization gathering script that would pump utilization stats into job-list via annotations (flux job annotate)? Like flux job annotate $JOBID mynoderank mynodesdata.

Then on the client side look at info via flux-jobs? Looking via flux-jobs would probably only be useful if jobs are like 1-2 nodes max. A post-processing script could aggregate the data a bit more for prettier output?