plugin: enforce max resource limits per-association

[cmoussa1]

Problem

The priority plugin tracks resource usage across an association's set of running jobs, but it does not enforce any limits
on this resource usage in the event where an association goes over their limit.

---

This PR looks to add limit enforcement on an association's set of running jobs according to their resource usage, specifically with ncores and nnodes. In job.state.depend, the jobspec for a submitted job is unpacked and its job size is calculated. If the job would put the association over either their max_cores or max_nodes limit, a dependency is added to the job and it is held until a currently running job completes and frees the appropriate amount of resources.

Since the priority plugin is now enforcing two different kinds of limits (max-running-jobs and max-resources), it needs a way to keep track of which dependencies a job has since now it can have one or both. A map is added to the plugin where the key is a the ID of a held job and the value is a list of dependencies associated with the job. A number of helper functions are added to search for held jobs' dependencies and remove them from the internal map when the job no longer has any dependencies on it.

In job.state.inactive, the process of releasing a held job is reworked to check if the job satisfies all requirements in order to be released. It is first checked to see that the association is under their max-running-jobs limit and then checked to see that the held job would not put the association over either their max_cores or max_nodes limit. The dependencies for the job are removed as they are satisfied, i.e if a job would satisfy the max-running-jobs dependency but not the max-resources-dependency, only the max-running-jobs dependency is removed and the job continues to be held. Once all requirements are met, the rest of the remaining dependencies are removed from the job and it can be released to run.

To avoid becoming its own mini scheduler, the priority plugin will only look at the first held job for the association. So, if an association has two held jobs, only the first held job will be considered for release.

I've expanded the test file that originally just tracked resources across an association's set of jobs to also test checking adding and removing dependencies. Different scenarios are walked through to make sure an association's set of jobs have the correct dependencies added to it and are removed correctly, such as having a job only have a max-running-jobs or max-resources dependency added to it, have both dependencies added to it at the same time, only have one of the two dependencies removed from a held job, etc.

[cmoussa1]

@ryanday36 when you get the chance, could you give this PR a high-level look and let me know if this looks good from your side? I think I'd mostly be curious if the way that the plugin is enforcing the resource limits looks good to you.

[ryanday36]

This looks like a good approach to me @cmoussa1. I'm not sure I understand where b->curr_nodes and b->curr_cores get updated. Is that already happening when jobs start / end? or is that still to come?

[cmoussa1]

Thanks @ryanday36. The cur_nodes and cur_cores for a job will get updated when the job enters RUN state and INACTIVE state. Sorry that wasn't made immediately clear - I added that support in #561.